🔍 Read the full analysis: What’s Behind Microsoft’s Claim That AI Outperforms Doctors? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Microsoft’s peer-reviewed study reports that its MAI-DxO system diagnosed difficult medical cases more accurately than generalist physicians in simulated tests. The results do not show that the system is better than specialists or care teams, or that it improves outcomes for real patients.
Microsoft’s MAI-DxO diagnostic system scored higher than generalist physicians on difficult medical cases in a study published October 22 in the peer-reviewed journal Science. The researchers reported 83.2% accuracy on 304 complex cases, compared with 36.1% for doctors using textbooks and the internet and 22.3% without those resources; the test used historical case records, not patients receiving care.
The paper, titled Sequential diagnosis with language models, also tested 325 emergency department cases. Microsoft reported that MAI-DxO reached 80.3% accuracy on that set, compared with 46.8% for physicians. The figures are results reported by the study’s authors, all of whom are affiliated with Microsoft. Publication in Science means the paper underwent peer review, but it does not establish that outside researchers have independently reproduced the results.
The benchmark presented each system with a short case summary. Participants then had to request additional information, such as questions or tests, from a simulated gatekeeper holding the full case record. The system was designed to combine language models in a panel that proposes diagnoses, selects next steps and challenges the group’s reasoning. The benchmark scored both the diagnosis and the estimated cost of gathering information.
The comparison was between the AI system and individual primary care doctors or hospital generalists, not specialists or physician teams. In the earlier preprint’s description, doctors could not use search engines or language models in one test arm; the published study also reported a group using textbooks and the internet. The cases were selected for difficulty and supplied as text. There was no physical examination or real-time imaging, and no patients were treated based on the system’s recommendations.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
What the Scores Mean for Care
The findings suggest that language models may help organize diagnostic reasoning and decide which information to seek, but they do not establish that MAI-DxO can safely replace a clinician. A high score on a set of difficult cases is not the same as better outcomes in hospitals, where patients may have incomplete or changing symptoms and clinicians must weigh risks, costs and preferences.
The benchmark’s cost estimates also need careful interpretation. According to the source account, tests were priced using billing codes and a 2023 U.S. hospital price list, alongside a fixed physician-visit estimate. Those figures are approximations, not a demonstration of savings for a health system or patient. The results may interest hospitals and insurers considering decision-support tools, but the study does not show how the system performs in ordinary clinical workflows or how often it could prompt unnecessary testing.
AI diagnostic software for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Microsoft Built the Benchmark
The study addresses a different task from a medical licensing-style test. Instead of giving a system every fact and asking it to choose a diagnosis, the Sequential Diagnosis Benchmark requires it to decide what to ask or test next. That makes the process of collecting information part of the evaluation, not just the final answer.
The source account says the benchmark draws on 304 complex cases from the New England Journal of Medicine and a separate set of 325 emergency department cases. It also notes that a June 2025 version appeared as a non-peer-reviewed preprint and reported different headline results, using an earlier model. Those earlier figures should not be treated as a direct comparison with the published paper, whose reported model and results have changed.
One result complicates the claim that the orchestration system itself accounts for the performance. The source says GPT-5 alone scored 78.9% on the complex-case set, close to MAI-DxO’s 83.2%. That suggests much of the score may come from the underlying model, while Microsoft’s multi-agent approach adds a smaller improvement. The study’s cost estimates and benchmark design are also part of the authors’ evaluation, not evidence of real-world clinical savings.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft CEO of AI Mustafa Suleyman, speaking to TechCrunch
medical diagnosis decision support tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits Beyond the Case Records
The study does not establish how the system performs with real patients, physical examinations, live imaging or the range of common conditions seen in routine care. The source account says the complex cases contain no healthy patients or harmless explanations, meaning the benchmark could not measure false alarms. A system tuned to pursue rare diagnoses may score well on a collection of unusually difficult cases while generating unnecessary concern or testing elsewhere.
Other details also warrant caution. The source says a model judged the answers against published diagnoses, with physician review showing good but imperfect agreement in a sample. When a requested test result was absent from a record, the gatekeeper generated a synthetic result. The full Science article is behind a paywall, so some methodological details described in the source come from the earlier preprint and could differ from the final paper. Independent replication and direct clinical evidence have not been provided in the supplied material.
It is also unclear whether the reported accuracy advantage would persist against specialists, teams of clinicians or doctors using the same tools available in practice. The comparison tested generalists working alone, while difficult cases may receive specialist or team review in real care.
As an affiliate, we earn on qualifying purchases.
A Prospective Hospital Study
Microsoft’s next described step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients will interact with the system alongside their usual care. That setting could provide evidence about how the tool works in a clinical environment, but the source does not give a start date, study size, evaluation measures or results.
Readers should look for information on whether the system changes clinicians’ decisions, how often its suggestions are accepted or rejected, and whether it affects diagnostic accuracy, patient outcomes, cost or safety. Until such evidence is reported, the Science results remain a benchmark comparison—not proof that AI outperforms doctors in clinical practice.
AI medical diagnosis training tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
Did Microsoft show that its AI is better than doctors?
Microsoft’s study reported higher diagnostic accuracy for MAI-DxO than for individual generalist physicians on selected historical cases. It did not test the system against specialists or physician teams, or establish superiority in treating real patients.
What cases were used in the study?
The study reported results on 304 complex cases from the New England Journal of Medicine and 325 emergency department cases. The evaluation used text-based case records in a simulator rather than live clinical encounters.
Was the study peer-reviewed?
Yes. The paper was published in Science on October 22, 2026. Peer review evaluates a paper’s methods and reporting; it does not mean the results have been independently reproduced.
Does the study show the AI saves money?
No. The benchmark included estimated test costs, but those estimates used a historical hospital price list and other assumptions. The results do not show actual savings for hospitals or patients.
What is the next test of the system?
The source account says Microsoft plans a prospective study at Beth Israel Deaconess Medical Center, where consenting patients will use the system alongside usual care. Details such as the study’s timing, size and outcome measures were not provided.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
