Human Oversight: What Agents Should Never Do Alone

The defining risk of AI agents is autonomous action. Establish clear boundaries:

Require human confirmation before:

• Writing to the clinical record

• Sending any patient communication

• Changing, cancelling, or creating appointments

• Submitting insurance claims or pre-authorizations

• Transmitting data to any external party

• Deleting or modifying existing records

The framework: Reading is lower risk than writing. Drafting is lower risk than sending. Suggesting is lower risk than deciding. Calibrate oversight to the consequence of error.

An agent that drafts a treatment plan explanation for dentist review is a productivity tool. An agent that sends treatment recommendations directly to patients is a liability.

Auditability: Knowing What the Agent Did

When software acts on behalf of your practice, you need to know what it did. This is both a HIPAA requirement (audit controls) and a practical necessity.

Your agent deployment should log:

• What patient data the agent accessed, and when

• Which tools and APIs it called

• What information it transmitted, and to whom

• What actions it took (messages sent, records modified, appointments changed)

• Authentication events (who authorized the agent's actions)

Ask your vendor:

• Are audit logs available to you, or only to the vendor?

• How long are logs retained?

• Can logs be exported for your own records?

• Who at the vendor can access logs containing your PHI?

If you cannot answer "what did the agent do last Tuesday?" you have an audit gap.

Retention, Training Data, and Secondary Use

What happens to patient information after the agent processes it? This question has more layers than most practices realize:

• Prompt retention: Are your inputs stored? For how long? Who can access them?

• Output retention: Are the agent's responses logged? Where?

• Model training: Is your data used to train or fine-tune the vendor's models? Can you opt out? Is opt-out the default?

• Vendor personnel access: Under what circumstances can the vendor's employees view your data?

• Memory and context: Does the agent retain information across sessions? Can that memory be inspected, corrected, or deleted?

• Downstream sharing: Is data passed to subprocessors, analytics providers, or integration partners?

Get written answers to each question. "We take privacy seriously" is not an answer.

Prompt Injection: The Agent-Specific Threat

Traditional cybersecurity focuses on keeping unauthorized people out. AI agents introduce a different vector: unauthorized instructions hidden in content the agent legitimately reads.

An agent that processes patient emails, reads web pages for drug information, or parses intake forms can encounter prompt injection — malicious instructions embedded in otherwise normal content. Examples:

• An email containing hidden text instructing the agent to forward patient data to an external address

• A webpage with invisible instructions to modify the agent's behavior

• An intake form field containing instructions disguised as patient information

The more authority an agent has, the more damaging a successful injection becomes. This is not hypothetical — the National Institute of Standards and Technology (NIST) has identified prompt injection as a key AI risk category in its Generative AI Profile, and 2026 federal initiatives are developing agent-specific security controls ^[2].

Mitigations:

• Limit agent permissions to the minimum required (see above)

• Require human confirmation for consequential actions (see above)

• Treat external content as untrusted input, not instructions

• Keep agents updated — vendors are actively developing injection defenses

• Include prompt injection in your security risk analysis