AI Panel Outperforms Doctors in Complex Medical Diagnosis Study

An artificial intelligence system developed by Microsoft correctly diagnosed complex medical scenarios with 83% accuracy in a simulation, more than doubling the 36% success rate of generalist physicians. The findings, published in the journal Science this week, highlight the growing capability of AI to handle intricate diagnostic challenges that often stump individual doctors. Mustafa Suleyman, the CEO of Microsoft AI, described the results as a significant step toward achieving what the company terms “medical superintelligence.”

The system, known as MAI-DxO, operates by coordinating multiple AI agents that function similarly to a panel of medical professionals. These agents work together to question patients, order necessary tests, and debate potential diagnoses. The study utilized complex cases from the New England Journal of Medicine to test the system. It is important to note that such cases are typically managed by specialists or team-based care in real-world settings, and these groups were not included in the direct comparison with the AI.

Microsoft’s results place the company alongside Google and OpenAI, which have recently claimed similar diagnostic capabilities. However, the study’s data indicates that much of the improvement stems from the underlying power of the AI models themselves, with the orchestration system providing additional benefits. Tests using models from various companies, including Anthropic and Meta, showed that the best results were achieved using OpenAI’s GPT-5 and GPT-5.5 models.

Cost and accuracy were both measured in the evaluation. When using GPT-5 with a basic prompt, the system scored 78.9% at a cost of approximately $8,400 per case. The DxO orchestration reduced this cost by about $3,000 while increasing accuracy by four points. For newer GPT-5.5 models, the system improved accuracy on complex cases from 72% to nearly 87% at a slightly lower cost. However, the benefits were less pronounced for emergency department cases, where accuracy remained around 85% regardless of the orchestration.

The system proved most useful for less capable models, such as Gemini 2.5 Flash, which saw its accuracy jump from 46.4% to nearly 69% with the assistance of DxO. Microsoft researchers noted that accuracy gains naturally diminish as models approach their theoretical limits. Suleyman stated that the company is collaborating with the Mayo Clinic to develop a healthcare foundation model, with a release planned for next year. He expressed confidence that this work will lead to major breakthroughs in processing medical records by 2027.

To validate these capabilities in a real-world setting, Microsoft is partnering with Beth Israel Deaconess Medical Center in Boston. This upcoming study will evaluate the clinical reasoning of DxO across various hospital environments, including emergency rooms and urgent care centers. Patients will receive standard treatment, while consenting participants will also interact with the AI system. Researchers will then compare the AI’s recommendations against those of human doctors. Suleyman believes this approach will provide compelling evidence for the future deployment of such systems at scale, aiming to make Microsoft the leader in medical clinical decision support tools.