Software validation and vibes: are we still “in control”?
The development and validation of software in healthcare is a complex process that is becoming increasingly important, especially with the arrival of the MDR (Medical Device Regulation) and the “covenant for the safe application of medical technology”. Recently, the “Guideline Validation of Software as a Medical Aid” was published to help doctors validate software after purchase up to and including putting it into use. Together with Paul, my colleague from the working group that drew up this guideline, I look back on the process and where we stand now.
At the time, I put together a panel of doctors myself and, together with them, dove into the topic on the basis of literature and engaged expertise on digital strategy. Besides valuable insights for the working group, open questions emerged here. For example: how important is clinical expertise in software development?
Our outdated digital structures often do not deliver sustainable, safe or reliable data for validation, which means that data curation remains intensive and costly. How do we ensure continuous updates of prediction models with prospectively collected data? How do we safeguard interests in the collaboration between doctors and developers? And how do we maintain visibility of clinical risks? There remains much to explore outside the scope of this guideline.
Validation vacation doctors?
According to Paul, clinical work will more often have to be interrupted for software validation, a “validation vacation”. The guideline helps with this, but the pace of new software and updates makes it more complex. Even small adjustments in the EHR take hours to validate.
Paul remembers how he used to program every piece of code, so that he knew exactly how everything worked. Nowadays it is different. Instead of struggling for hours ourselves, we now take tools from digital libraries. You call up those tools and let them do the work, until it meets your wishes. This “vibe-coding” offers convenience: the AI agent solves problems quickly with access to worldwide solutions. But this convenience raises questions about control and reliability.
“Wappie-software” validation?
Paul compares trust in software to a “wappie message”: it looks logical, but somewhere there is a wrong turn in the reasoning. With software it is difficult to find such a hidden defect. Validation checks whether software works as expected, but more important is whether, in the event of deviations, software also warns you with an error message. In this way Paul himself discovered an error in a cytostatics dosage that the software did not notice. Tools from online libraries raise additional questions: have they been validated, by whom, and under which conditions?
False sense of control
The guideline suggests control over software validation, but in reality risks, build-up and defects are often unknown. This can lead to health risks without doctors being aware of them. Or worse: deliberately deployed software risks in the interest of a malicious individual or group!
According to Paul it is no different than before: you may still expect that the developer guarantees sound and safe products. And the same applies to doctors. They are the ones who must interpret where risks lie and when warnings are needed.
Exploring outside the scope is unavoidable!
The guideline emphasizes the importance of a broader view of the entire life cycle of software. Doctors must be involved integrally, in order to contribute their expertise—something that developers, policymakers, lawyers and other domain experts do not take responsibility for! Doctors must also ensure that legislation, frameworks, standards or norms continue to deliver added value for health.
Doctors will increasingly immerse themselves in relevant domains such as ICT, law and policy in order to provide crucial input. See it as an LLM prompt: the more specifically formulated, the more usable the answer. But also letting the LLM go at the right moment: an art, because you catch insights with it that you otherwise would never have come up with yourself. And let that be a piece of cake, because letting that connection be made with the patient is core business for the doctor!
References
1.
2. A Large Language Model (LLM) is an advanced AI model that is trained on enormous amounts of text to understand and generate human language. It can write texts, answer questions and conduct conversations based on context and patterns.
Gabriëlle Speijer is a radiation oncologist at Haga Hospital and founder of CatalyzIT. Paul de Wolf is a hospital pharmacist and Chief Pharmacy Information Officer at Haga Hospital.
