№ 33RiskCanonRelevant to: Manufacturer · Clinician Corpus reviewed April 27, 2025

    “LLM clinical copilots are safe because we put a disclaimer in the system prompt.”

    A widely held belief in MedTech, and a dangerous one. Here's what the regulators, the standards bodies, and the field evidence actually say.

    The Myth

    LLM clinical copilots are safe because we put a disclaimer in the system prompt.

    This belief shows up in boardrooms, design reviews, and FDA pre-submissions every week. It feels safe. It is not.

    The Reality

    Indirect prompt injection through a patient note, lab PDF, or EHR field can override the system prompt, exfiltrate context, or rewrite recommendations. Disclaimers do not constrain model behavior. OWASP LLM01 and FDA's emerging AI guidance both treat this as a security-and-safety issue.[1]

    Why it matters

    A clinical copilot that follows attacker-controlled text in a chart can recommend wrong dosing or leak PHI to an external endpoint, neither of which a disclaimer prevents.[2]

    OWASP LLM01Prompt injection is the #1 LLM application risk
    What to do instead

    Concrete next steps.

    Specific, defensible actions to replace the misconception with practice. Each item maps back to the citations above.

    1. 1

      Treat all retrieved or user-supplied text as untrusted input

      Patient notes, lab PDFs, and EHR fields can carry indirect prompt injection. Sanitize, isolate, and never let untrusted text alter system instructions.

    2. 2

      Constrain model output with structured contracts

      Use function calling, JSON schemas, or grammars to bound outputs. Free-text disclaimers are not a control.

    3. 3

      Add an output-side safety classifier

      Pass model output through a separate classifier that flags unsafe recommendations, PHI exfiltration patterns, and policy violations before showing or acting on them.

    4. 4

      Log prompts, retrievals, and outputs for incident investigation

      Without observability you can't reconstruct an injection attack. Store enough context (with PHI controls) to investigate suspected misbehavior.

    References

    The receipts.

    Review the sources behind this entry. Each reference shows which claim it supports and who published it.

    2 sources
    1. 1
      OWASP Top 10 for LLM Applications
      Supports: The RealityIndustry / other owasp.org
    2. 2
      NIST AI Risk Management Framework
      Supports: Why it mattersRegulator / government nist.gov

    Found a broken or outdated source? See our evidence methodology and send a correction.

    Need help getting this right?

    Blue Goat Cyber works with MedTech manufacturers on premarket and postmarket cybersecurity programs.

    Talk to an expert
    Related misconceptions

    If this one bit you, these will too.

    Inferred from shared tags and category — not editorial picks.