Question Answering On Medqa Usmle

Metrics

Accuracy

Results

Performance results of various models on this benchmark

Model Name	Accuracy	Paper Title	Repository
Flan-PaLM (540 B)	67.6	Large Language Models Encode Clinical Knowledge	-
Meditron-70B (CoT + SC)	70.2	MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
GAL 120B (zero-shot)	44.4	Galactica: A Large Language Model for Science
Med-Gemini	91.1	Capabilities of Gemini Models in Medicine	-
VOD (BioLinkBERT)	55.0	Variational Open-Domain Question Answering
Shakti-LLM (2.5B)	60.3	SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments	-
Med-PaLM 2 (5-shot)	79.7	Towards Expert-Level Medical Question Answering with Large Language Models
GrapeQA: PEGA	39.51	GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering	-
BioLinkBERT (base)	40.0	LinkBERT: Pretraining Language Models with Document Links
DRAGON + BioLinkBERT	47.5	Deep Bidirectional Language-Knowledge Graph Pretraining
LLAMA-2 (70B)	59.2	MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Med-PaLM 2 (CoT + SC)	83.7	Towards Expert-Level Medical Question Answering with Large Language Models
GPT-4	90.2	Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
MedMobile (3.8B)	75.7	MedMobile: A mobile-sized language model with expert-level clinical capabilities	-
Med-PaLM 2	85.4	Towards Expert-Level Medical Question Answering with Large Language Models
BioLinkBERT (340 M)	45.1	Large Language Models Encode Clinical Knowledge	-
OPT (few-shot, k=5)	22.8	Galactica: A Large Language Model for Science
Meerkat-7B (Single)	70.6	Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks	-
PubMedGPT (2.7 B)	50.3	Large Language Models Encode Clinical Knowledge	-
BioBERT (large)	36.7	BioBERT: a pre-trained biomedical language representation model for biomedical text mining

0 of 27 row(s) selected.