Microsoft's MAI-DxO Diagnosed 83% of Hard Cases in a Science Study, Against 36% for Generalist Doctors, in a Text-Only Benchmark
A peer-reviewed paper in Science from Microsoft researchers reports that its Diagnostic Orchestrator, running on GPT-5, reached the correct diagnosis in 83.2% of 304 New England Journal of Medicine case records and 80.3% of 325 emergency admissions. Generalist physicians in the study scored 36.1% and 46.8%. The authors stress the results come from a controlled, text-only benchmark, not clinical practice; all authors are Microsoft employees.
Peer review in Science gives Microsoft's diagnosis results more weight than a company preprint, but what it validates is a benchmark: text-only cases, generalist doctors and no patients. The paper, "Sequential diagnosis with language models," appeared in Science's October 22 print issue, and Reuters and The Next Web reported on it on Friday, October 23.
What the study tested
The researchers built a benchmark, SDBench, from 304 New England Journal of Medicine clinicopathological case records and 325 emergency-department-to-admission cases from the MIMIC-IV-ED dataset. Each case starts from a short presentation, and the diagnostician, either the AI system or a physician, must decide which questions to ask and which tests to order before committing to a diagnosis. The paper scores both accuracy and the estimated cost of tests ordered.
The AI system, which the paper calls the Diagnostic Orchestrator (DxO) and The Next Web, citing Reuters, calls the MAI Diagnostic Orchestrator (MAI-DxO), simulates a panel of physicians with different roles that revise the diagnosis as results come in.
The results
According to the paper:
- Hard case records. Running on OpenAI's GPT-5 with no budget limit, DxO was correct on 83.2% of the case-record challenges at an average estimated test cost of $5,431. Generalist physicians averaged 36.1% with search and reference tools, and 22.3% without.
- Emergency admissions. DxO reached 80.3%, against 46.8% for physicians.
- Lower budgets. With budget limits, DxO's accuracy on the case records fell to 79.6% at an average of $1,303.
- Newer models. The authors say they repeated the evaluation with GPT-5.5 before publication and that DxO's advantage over a plain prompt grew.
Dominic King, a corporate vice president in Microsoft AI's health unit, told Reuters, as quoted by The Next Web, that medical AI has largely been judged on multiple-choice tests: "No patient ever came to see me with a nice summary." The Next Web reported that Microsoft plans to work with Beth Israel Deaconess Medical Center in Boston to test whether the system can help triage patients in emergency waiting rooms, and that it is not yet used in clinical practice.
Limits
- The authors write that their findings "were obtained in a controlled benchmark setting." They say the comparisons "do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice."
- The physicians were generalists working alone, and the case mix excludes healthy or benign visits. Cases were delivered as text, with no imaging, waveforms or pathology slides. The paper says physical-exam findings could be requested as text; The Next Web describes the study as having no access to physical examinations.
- Test costs are estimates from one US hospital's published price list, not real bills.
- All authors are Microsoft employees who hold equity in the company, and eight are named inventors on a related Microsoft patent application, according to the paper's competing-interests statement.
- We read The Next Web's account of the Reuters report but not the Reuters report itself.