LLM clinical copilots are safe because we put a disclaimer in the system prompt.
This belief shows up in boardrooms, design reviews, and FDA pre-submissions every week. It feels safe. It is not.
A widely held belief in MedTech, and a dangerous one. Here's what the regulators, the standards bodies, and the field evidence actually say.
LLM clinical copilots are safe because we put a disclaimer in the system prompt.
This belief shows up in boardrooms, design reviews, and FDA pre-submissions every week. It feels safe. It is not.
Indirect prompt injection through a patient note, lab PDF, or EHR field can override the system prompt, exfiltrate context, or rewrite recommendations. Disclaimers do not constrain model behavior. OWASP LLM01 and FDA's emerging AI guidance both treat this as a security-and-safety issue.[1]
Why it matters
A clinical copilot that follows attacker-controlled text in a chart can recommend wrong dosing or leak PHI to an external endpoint, neither of which a disclaimer prevents.[2]
Specific, defensible actions to replace the misconception with practice. Each item maps back to the citations above.
Patient notes, lab PDFs, and EHR fields can carry indirect prompt injection. Sanitize, isolate, and never let untrusted text alter system instructions.
Use function calling, JSON schemas, or grammars to bound outputs. Free-text disclaimers are not a control.
Pass model output through a separate classifier that flags unsafe recommendations, PHI exfiltration patterns, and policy violations before showing or acting on them.
Without observability you can't reconstruct an injection attack. Store enough context (with PHI controls) to investigate suspected misbehavior.
Review the sources behind this entry. Each reference shows which claim it supports and who published it.
Found a broken or outdated source? See our evidence methodology and send a correction.
Need help getting this right?
Blue Goat Cyber works with MedTech manufacturers on premarket and postmarket cybersecurity programs.
Inferred from shared tags and category — not editorial picks.
AI/ML medical software doesn't need new security thinking.
Matched on 1 shared tag.
Matches are inferred mechanically — no editorial picks. Each related myth gets a score of 2 points per shared tag plus 1 point if it sits in the same category. The top three with score ≥ 2 are surfaced.
Our AI model is ours, so we control its supply chain.
Matched on 1 shared tag.
Matches are inferred mechanically — no editorial picks. Each related myth gets a score of 2 points per shared tag plus 1 point if it sits in the same category. The top three with score ≥ 2 are surfaced.
A Predetermined Change Control Plan lets us update our AI model whenever we want.
Matched on 1 shared tag.
Matches are inferred mechanically — no editorial picks. Each related myth gets a score of 2 points per shared tag plus 1 point if it sits in the same category. The top three with score ≥ 2 are surfaced.