AI can be wrong. The product has to be designed for that.
Menaela is built on the assumption that language models can misunderstand, hallucinate, receive incomplete context and call the wrong thing. Safety comes from how those failures are contained — not from pretending the model will never make them.
The model may interpret the request and choose an allowed capability. The clinic application still decides what data it can see, what action is valid, and whether that action actually happened.
These are not theoretical “AI ethics” categories. They are ordinary failure modes that appear when a conversational agent is connected to real clinic operations.
The AI invents a clinic fact
Example: A patient asks for a price, preparation instruction or service detail that is not configured, and a language model could generate something that sounds plausible.
How Menaela handles it: Menaela is instructed to answer clinic-specific facts from configured clinic context and tools. If reliable information is not available, the valid outcome is to say it cannot confirm the answer and route the question to staff — not to fill the gap from general knowledge.
The AI offers a slot that is no longer available
Example: A patient sees 16:00 in the conversation, but another booking reaches the calendar before they confirm.
How Menaela handles it: The language model does not own availability. The deterministic scheduling layer recalculates and validates the exact requested time immediately before booking or rescheduling. A stale conversational suggestion cannot force an invalid appointment into the schedule.
The patient asks a medical question the agent should not answer
Example: “Is this symptom dangerous?”, “Should I stop this medication?”, “What does my result mean?”, or “Am I suitable for this treatment?”
How Menaela handles it: Patient-specific diagnosis, treatment, medication, risk, result interpretation, suitability and urgency are outside the autonomous patient toolset. Those requests are escalated to the clinic rather than converted into medical advice. Emergency-like wording receives concise urgent-care guidance in addition to escalation.
The wrong patient context is used
Example: A conversation contains another person’s name or an internal identifier and the model tries to follow it.
How Menaela handles it: Patient and clinic identity are server-bound. The model receives the conversation inside an already established tenant and person context and cannot switch to an arbitrary patient simply because free text asks it to.
The model misunderstands an ambiguous request
Example: “Move me to Friday” could refer to more than one appointment, or “book me with Dr Cohen” could still be missing the service.
How Menaela handles it: Ambiguity is treated as missing input, not an invitation to guess. Menaela resolves entities against clinic data and asks a targeted clarification or confirmation before a consequential action is committed.
A tool or integration fails mid-conversation
Example: WhatsApp is working, but the booking operation times out or an external service returns an error.
How Menaela handles it: Tool failure is surfaced as failure. The assistant should not claim that an appointment was booked, a message was sent or a record was changed unless the application confirms the operation. Staff can take over the thread with the context preserved.
AI-generated clinical documentation is wrong
Example: A transcript is incomplete or the draft note contains a mistaken interpretation.
How Menaela handles it: Scribe output remains a draft. The clinician reviews, edits and signs the note; the model does not silently promote generated text into an authoritative signed record.
A personal WhatsApp contains conversations the clinic AI should never touch
Example: A solo practitioner connects the WhatsApp number they already use. Patients write there, but so do friends, family or unrelated contacts.
How Menaela handles it: WhatsApp autonomy can be restricted to known patients only. Sender identity is checked before the autonomous agent is invoked; unknown contacts receive no AI reply and are not automatically promoted into leads. Manual WhatsApp use remains available to staff.
The AI has more power than the surface requires
Example: A patient-facing WhatsApp assistant should not have the same capabilities as an authenticated clinic administrator.
How Menaela handles it: Autonomy is scoped by execution surface. Patient conversations receive a deliberately narrower tool catalog and context than internal staff agents. The model can only call operations explicitly exposed for that role and workflow.
No single prompt is the safety system. The controls sit across identity, retrieval, tools, deterministic application logic and human escalation.
1. Constrain context
Give the agent the clinic, user, patient and operational context required for the current task — not unrestricted access to everything.
2. Constrain tools
Expose explicit application operations rather than raw database access. Patient-facing and internal agents receive different capability sets.
3. Keep business rules deterministic
The model interprets language; the application remains authoritative for permissions, availability, validation and persistence.
4. Confirm consequential actions
When an action is ambiguous or materially changes clinic data, the flow can require clarification or confirmation before execution.
5. Escalate unsupported judgement
Medical judgement, policy exceptions and uncertain cases move to clinic staff instead of being improvised by the model.
6. Preserve an audit trail
Agent-originated actions can be traced with actor, tenant, operation and target context so automation remains inspectable.
- Clinic-specific answers require configured clinic information.
- Patient-specific diagnosis, treatment, medication, risk and result interpretation are escalated.
- Emergency-like messages receive concise urgent-care guidance plus escalation.
- Patient-facing tools cannot inherit unrestricted staff capabilities.
- External autonomy can be enabled per source; WhatsApp can be restricted to known patients before the model is invoked.
Safety is about what happens when the AI is uncertain, wrong or asked to do something outside its authority. Security is about tenancy, permissions, architecture and protecting access to data. They overlap, but they are different problems. Read the security & architecture page →
Want to test the boundaries?
Bring us the awkward cases: ambiguous bookings, missing clinic facts, medical questions, identity edge cases and tool failures. Those are exactly the scenarios an operational AI should be tested against.
Talk to us