Stop Testing AI Against Doctors. Test Doctors With AI.

Health Tech Nov 9, 2026• By MANUEL PINILLA Every few months a new study reports that an AI model has outperformed doctors on a set of diagnostic cases. The latest one I’ve read came out in Science in October: Microsoft’s MAI-DxO, tested on 304 complex cases from the New England Journal of Medicine and run as step-by-step simulations, reached the correct diagnosis in about 83% of them. Primary care doctors and general internists, working on their own with reference materials, got 36%. Microsoft says the next step is a study with real patients. I build clinical AI, so these results matter to my work, but they answer a different question from mine. In the setting we work on, emergency departments and primary care, a doctor will be in the room, and that doctor signs the diagnosis and answers for it. While that is the case, a score of AI against doctors tells me little about whether a tool helps. What I need to know is whether the doctor, with the AI, gets the diagnosis right more often, and what happens when the AI is wrong. There is a second gap. A published case arrives already written up: the history is complete, the relevant findings are there and the question is clear. A real pati...