MedExamHelperThe AI Patient Realism Test: Role Drift, Information Leakage and Unrealistic Cooperation
A patient is not a chatbot with a name. A patient is a person who is frightened, vague, in pain, contradicting themselves, and not on your side of the desk. Building an AI that behaves like a real patient is the hardest part of the product we are building, and it is the part most shortcut. This post is about the three failure modes we test for, and the same three failure modes you should test for in any AI patient you evaluate.
We are not pretending we have solved all three. We are pretending it is worth being honest about them.
These three failure modes are not unique to us. They are the default behaviour of a general-purpose language model asked to play a patient. The model is trained to be helpful, to fill in details, to keep the conversation going, and to make the user feel competent. Every one of those defaults is the opposite of what a real patient does in an exam. The reason we test for them is that the defaults are strong, and they reassert themselves any time you let the guard down.
Failure mode one — role drift
The patient starts behaving like something other than a patient. They begin tutoring you. They begin asking you questions back. They volunteer the differential. They say things like "wouldn't it be useful to consider a pulmonary embolism here?" — which is what a senior would say, not what a 54-year-old with pleuritic chest pain would say.
Role drift happens because the underlying model is trained to be helpful, and a helpful patient is an unhelpful simulator. The model wants to teach. The patient is meant to withhold. The more capable the model, the more subtly it drifts: not into outright tutoring, but into small concessions — a clearer history, a tidier story, a moment of hesitation the model invented because real patients hesitate.
You spot role drift when the patient's sentences start to feel like notes. A real patient's answer to "can you tell me more about the pain" is messy. It goes back over ground, repeats itself, says something irrelevant. A simulated patient's answer is too neat. That neatness is the give-away.
Failure mode two — information leakage
The patient volunteers the diagnosis. Not always in the obvious way — they do not say "I think I am having an MI" — but in the cumulative weight of what they offer. They tell you their pain is crushing. They mention the radiation to the left arm. They volunteer the diaphoresis. They bring up the nausea. By the time you have asked three questions, you have the diagnosis, and you have not had to work for it.
The exam marker would never give you that. The exam marker gives you a 54-year-old who came in because their wife made them, who is not sure it is anything serious, who is a bit uncomfortable, and who would rather talk about the match. That is the level of withholding you need an AI patient to do. Information leakage flatters you. It also fails you, because the station stops testing the things it was designed to test.
Failure mode three — unrealistic cooperation
The patient never gets vague, never gets scared, never contradicts themselves, never refuses to answer, never mumbles. The patient is too compliant. They answer in full sentences. They always wait for the next question. They do not interrupt.
Real patients interrupt. They answer the wrong question. They look at their phone. They get upset when you ask about alcohol. They do not know the names of their medications. They say "the white one, round". They tell you about their aunt who had the same thing. They are late to the station, or their notes are missing, or the chart is illegible. An AI patient who never does any of these things is not an AI patient; it is a chatbot with a stethoscope.
The cost of unrealistic cooperation is the most subtle of the three. You feel like you are doing well. You feel fluent. You move through the station smoothly. You walk out confident, and the marking scheme says you missed the bit where you handled a difficult silence, the bit where you managed an interruption, the bit where the patient's answer contradicted the chart. The fluency was the lie.
What each failure mode costs you in a real exam
Role drift makes you practise the wrong skill. You are practising eliciting a tidy history from a tidy patient, which is not what the OSCE tests. Information leakage makes you over-confident; you score the diagnosis without doing the reasoning, and the reasoning is what fails under exam pressure. Unrealistic cooperation makes you fluent at the wrong register; the real patient is rougher than the simulator, and the gap shows on the day.
Each failure mode is a way the AI inflates your apparent performance. Each failure mode is a way the AI under-tests you.
How we test for it, and how you can
The patient is instructed never to volunteer more than your question asks. While you practise, a second model checks the patient's replies for slips. Before each release, scripted students try to lead, pressure and trick the patient, and a judged test measures how often it answers a different question or offers history nobody asked for. That test is re-run whenever the patient's model or instructions change. And every case and checklist is reviewed and approved by our clinician, Dr Manju Raja, before you can play it.
You can run your own check. Pick a case you have done before in a real exam. Compare the patient's behaviour with the AI's. If the AI is consistently tidier, kinder, more cooperative or more informative than the patient you remember, the AI is showing one of the three failure modes. You are not getting better. You are getting smoother.
The checklist before you trust an AI patient
Before you stake a station rehearsal on a tool, ask five things.
- Does the patient ever withhold relevant facts across more than three questions?
- Does the patient ever volunteer facts you did not earn?
- Does the patient ever contradict a previous answer, change their mind, or admit they forgot?
- Does the patient ever interrupt, refuse to answer, or ask you to repeat?
- Does the marking scheme score the things the patient withheld, or only the things they offered?
If two of those answers are no, the tool is a chatbot, not a patient.
The product we are building is not a finished answer to this. Every one of our 250+ stations, case and checklist, is reviewed and approved by a practising clinician before students can play it.
Try it
The fastest way to find out whether an AI patient avoids the three failure modes is to use one. Pick a case, run it, and watch for the moment where the patient behaves too well. That moment is the failure mode you would have missed in a real station.
A good AI patient should make you slightly uncomfortable in places you would not have been uncomfortable in a textbook case. The discomfort is the rehearsal. If the case is too comfortable, you have not rehearsed; you have been entertained. Real patients do not flatter, and a simulator that flatters is closer to a sales demo than to an exam.