Reading Group (+🧋): Measuring Physicians and AI Relevance Alignment with MedPAIR

Join the Snorkel AI Reading Group, a recurring forum to explore the latest frontier developments in AI while building meaningful connections within the community.

LLMs now beat the human average on standardized medical exams, but a right answer doesn't mean a model reasoned its way there correctly: It might have latched onto an extreme lab value, stray detail, or piece of context a physician would immediately discount, and still landed on the correct choice by accident.

In this session, Yuexing Hao (Microsoft, MIT EECS) will present her work that introduces MedPAIR: Medical Dataset Comparing Physicians and AI Relevance Estimation and Question Answering to catch exactly that gap.

Among other things, you'll learn:

• Why a model can answer a medical question correctly while relying on completely different - and sometimes spurious - information than a physician would, and why accuracy alone can't catch it.

• How MedPAIR's sentence-level annotation process surfaces exactly where physicians and LLMs part ways on what counts as clinically relevant.

• Why models often overweight superficial signals, like an unusually extreme test result, while missing subtler cues that trainees flagged as decisive.

• Across four medical QA benchmarks, how stripping out the context physicians deemed irrelevant lifted LLM accuracy, which in some cases was enough to beat the physicians' own average.

Agenda:
4 pm - doors open
4:30 pm - talk begins
5:30 pm - research discussion and networking

🧋🧋🧋 Boba tea and other refreshments will be provided ! 🧋🧋🧋

This work appeared as an Oral Presentation in the NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models. arXiv preprint available here.

Yuexing Hao is a Researcher at Microsoft and Postdoctoral Associate at MIT EECS Healthy ML Group. She received her PhD in Human-Centered Design from Cornell University.