- Safety is an application architecture problem, not a prompt-writing trick.
- Patient identity and tenant boundaries should be server-owned.
- Patient-facing agents should receive a narrower tool catalog than internal staff.
- Medical judgement, uncertain facts and failed operations need explicit escalation behavior.
Start with the assumption that the model can be wrong
Language models are probabilistic systems. Even very good ones can misunderstand a reference, overgeneralize from context or state something plausible that is not true for this clinic.
The product should therefore be designed so that a wrong sentence does not automatically become a wrong action.
Constrain context before you constrain language
A patient-facing assistant rarely needs the entire patient record. It needs the minimum context required for the current operation: perhaps the current appointment, an approved price, a service definition or a specific recorded-care instruction.
Reducing context lowers both privacy exposure and the number of irrelevant facts the model can accidentally misuse.
Give the patient agent fewer tools than the clinic manager
A staff user may be allowed to create patients, update services, change prices or inspect a broader record. A patient conversation should not inherit those capabilities simply because both use the same foundation model.
Execution surfaces should have different permission envelopes.
The model can only misuse capabilities it has been given.
A narrow patient-facing tool set is a stronger safety control than a paragraph saying “never do anything unsafe.”
Keep irreversible rules outside the model
The application should remain authoritative for availability, permissions, patient identity, persistence and signed record state.
If the model asks to book an invalid time, the booking operation should reject it. If it asks to access another patient, the server context should prevent it. If a note is signed, an AI rewrite should create an amendment or new version rather than silently altering history.
Medical knowledge is not medical authority
A patient may ask whether a symptom is dangerous, whether to stop a medication or whether a referral is sufficient. Some clinics may have approved administrative information around those questions; others require staff review.
The safest autonomous agent knows the difference between a clinic fact it can retrieve and a judgement it is not authorized to make.
One conversation, two different authority levels
Do I need a referral for the consultation?
The clinic’s policy says a referral is not required for a private consultation.
My chest pain is worse today. Is it okay to wait for Thursday?
I can’t assess whether it is safe to wait. I’m escalating this to the clinical team; if symptoms may be urgent, seek immediate medical help.
Tool failure must look like failure
If a booking call times out, the assistant should not infer success from the intent. It should verify the resulting state or report that the action could not be confirmed.
The same principle applies to messaging, file operations and updates. A confident sentence is not an audit trail.
Menaela’s safety model
Menaela separates AI reasoning from the clinic’s authoritative operations. Context is tenant- and identity-bound, tools are role-specific, business rules are enforced by the application and unsupported judgement is escalated.
It is not a promise that the model will never be wrong. It is a design intended to keep ordinary model error from becoming an uncontrolled clinic action.
