How to Evaluate an AI Receptionist for Your Clinic: 12 Tests Beyond the Demo

1. Ask for a real booking, then change the constraints

Start with something normal: "Dr Cohen Thursday after four." Then say "anything earlier?" Then "someone else is fine if they do the same service."

Watch whether the system preserves the constraints that still matter and changes only the one you changed.

2. Ask for an unavailable provider

A good system should not simply fail or silently substitute another doctor. It should follow the clinic’s rule: another day, another provider, waiting list, human handoff or some combination.

3. Ask a price question, then a payer question

These often look similar but have different authority. A configured clinic price may be safe to answer. Patient-specific coverage or eligibility may not be. See whether the product can tell the difference.

4. Mix an administrative request with a medical question

Patients will do this in production. The agent should complete what it can safely do and escalate the clinical judgement instead of answering everything because the model technically can.

5–8. Test identity, state, handoff and failure

These four tests tell you whether the product has an application behind the conversation.

• Identity: ask for information that should require patient identification and see what is disclosed before verification.
• State: change your mind several times and refer back with phrases like “the later one” or “same doctor.”
• Handoff: ask something the agent cannot answer and inspect what staff receive.
• Failure: make the booking or integration call fail and see whether the assistant claims it succeeded.

9–10. Test your language and your clinic knowledge

Do not accept an English demo if your patients write in Hebrew or Russian. Test local names, mixed-language messages, typos, daypart expressions and medical terms.

Then change one clinic fact — a price, provider availability or service rule — and find out how quickly a manager can update what the agent knows.

11. Ask what you can inspect after the conversation

You should be able to see the message history, the action result and ideally the reason an escalation occurred. An autonomous system without observability is very hard to improve safely.

12. Ask what the AI cannot do

This may be the most useful question in the entire evaluation. A serious product should have a clear answer.

If every boundary is described as "the AI can handle it," you are not hearing about the product architecture. You are hearing sales copy.