Microsoft's AI diagnostic panel beats doctors in a Science study

MAI-DxO — the Microsoft AI Diagnostic Orchestrator — does not ask one model for an answer. It turns a language model into a virtual panel of physicians: agents propose diagnoses, pick questions to ask or tests to order, and challenge the panel’s emerging view over several rounds of debate, updating their assessment as evidence arrives and weighing each test’s likely value against its cost.

To test it, Microsoft built a new benchmark, SDBench, because multiple-choice medical exams don’t resemble diagnosis: in practice clinicians start with limited information and build the picture step by step. SDBench converts 304 complex NEJM clinicopathological conference cases and 325 emergency-department admissions (from the MIMIC-IV-ED database at Boston’s Beth Israel Deaconess Medical Center) into interactive encounters where a separate AI “gatekeeper” releases information only when it’s requested, and every test ordered carries an estimated cost based on real CPT pricing. Microsoft says it is open-sourcing SDBench for select NEJM cases — benchmark, gatekeeper, judge, and cost-estimation code — so others can reproduce the evaluation and challenge it. Per Unite.AI, citing the SDBench project site, the code is MIT-licensed on GitHub, but the dataset derived from the NEJM cases requires a licence from NEJM Group, the MIMIC subset is distributed through PhysioNet with credentialing, and five fully synthetic cases ship as the default open dataset.

In Microsoft's own testing on the complex NEJM cases, MAI-DxO reached 83.2% diagnostic accuracy. Generalist physicians scored 36.1% with search and reference tools and 22.3% without, which Microsoft says makes MAI-DxO about 2.3 and 3.7 times as accurate. On the more routine emergency-department cases, the AI hit 80.3% against 46.8% for the physicians. Even the single best-scoring physician (51.8% on the NEJM set, 62.5% on the ED set) came in below DxO on both accuracy and cost. One caveat on the comparison: the AI scores cover all 304 NEJM and 325 ED cases, but the physician scores come from held-out test sets — 56 NEJM cases, per Unite.AI and the paper — so the two sides did not face identical case counts.

The main results ran on OpenAI’s GPT-5, and the gains persisted — and improved — on newer models including GPT-5.5 without changing the orchestration. TechCrunch, reading the study’s data, reports that the harness also lifted weaker models, moving Gemini 2.5 Flash from 46.4% to nearly 69%. Microsoft AI CEO Mustafa Suleyman is framing the work as a step toward “medical superintelligence” — surpassing the strongest human-led health systems at medical reasoning.

The AI and the physicians didn’t just score differently; they practiced differently. Physicians asked an average of 5.6 questions before ordering their first test. DxO asked 25.5 — far more extensive, structured history-taking before spending anything. When DxO was wrong, its errors were usually a closely related diagnosis at the wrong level of specificity; physician errors tended to land further from the correct answer.

Cost is where the story gets nuanced. Without budget limits, DxO’s workups cost more than the physicians’: $5,431 vs. $4,725 per NEJM case, $4,979 vs. $3,400 per ED case. But when the researchers imposed budgets, DxO cut costs while staying above the physician average — 79.6% accuracy at $1,303 per NEJM case, 73.2% at $2,634 on the ED set. Across both datasets, no other human or AI approach was both more accurate and less costly.

This was a controlled, text-only benchmark — not real clinics. The human comparison was against individual generalists and hospitalists working alone, not specialists or teams; the authors say they expect the gap to narrow against specialists matched to each case. The NEJM cases are chosen for teaching value and often include specialists’ conclusions. There was no imaging, no waveforms, no forgetful patients — and every case ended in a pathological diagnosis, so benign visits were excluded and the mix doesn’t reflect routine practice. The study measured the diagnostic phase only: not treatment decisions, patient safety, or outcomes.

All authors are Microsoft employees who hold equity in the company, and eight are named inventors on a Microsoft patent application for the evaluation framework. TechCrunch notes that much of the accuracy lift traces to the underlying model, not the orchestration layer itself — on GPT-5, Microsoft’s own figures show the harness lifting accuracy by about four points (78.9% to 83.2%) while cutting estimated cost by about $3,000 per case (roughly $8,400 to $5,400). Microsoft says it plans a prospective study with Beth Israel Deaconess Medical Center in which patients interact with the system directly; no firm timeline has been given, though Dominic King told The Verge it could “get underway in the next couple of months.” Separately, Microsoft and Mayo Clinic announced in June that they are developing a healthcare-specific frontier model that Mayo will own; no release date has been announced. As King, Microsoft AI’s VP of Health, told TechCrunch: “The biggest challenge in healthcare delivery at the moment is the growing mismatch between demand and supply of medical expertise.”