MMLU Pro
Academic multiple-choice benchmark covering 14 subjects including STEM, humanities, and social sciences.
Systems
| Model | Accuracy | Cost In / Out | Latency |
|---|---|---|---|
| Gemini 3.1 Pro Preview (02/26) | 90.99%± 0.28 | $2 / $12 | 23.91s |
| Gemini 3 Pro (11/25) | 90.10%± 0.30 | $2 / $12 | 24.27s |
| Claude Opus 4.7 | 89.87%± 0.31 | $5 / $25 | 187.83s |
| Claude Opus 4.6 (Thinking) | 89.11%± 0.45 | $5 / $25 | 49.45s |
| Gemini 3 Flash (12/25) | 88.59%± 0.32 | $0.5 / $3 | 36.25s |
| Claude Opus 4.1 (Thinking) | 87.92%± 0.33 | $15 / $75 | 27.67s |
| Qwen 3.6 Plus | 87.67%± 0.32 | $0.5 / $3 | 46.18s |
| GPT 5.4 | 87.48%± 0.42 | $2.5 / $15 | 28.88s |
| Claude Sonnet 4.5 (Thinking) | 87.36%± 0.39 | $3 / $15 | 38.20s |
| Claude Sonnet 4.6 | 87.34%± 0.43 | $3 / $15 | 55.41s |
Key Takeaways
- Gemini 3.1 Pro Preview (02/26) leads the benchmark with 90.99% accuracy. Gemini 3 Pro (11/25) follows in second place at 90.10%, followed by Claude Opus 4.7 at 89.87%.
- It’s clear that this benchmark is largely saturated - there will be diminishing utility to model providers continuing to hill climb on MMLU Pro.
Dataset and Context
MMLU is a commonly-used academic evaluation that tests models on multiple-choice questions on subjects like physics, chemistry, etc. MMLU Pro is an improved version of MMLU, focusing on data quality and diversity. MMLU Pro consists of over 12,000 question-answer pairs, created through:
- Filtering and enhancing classic MMLU questions
- Expanding answer options from 4 to 10 choices
- Incorporating new sources (STEM Website, TheoremQA, SciBench)
- Expert verification of questions
The benchmark evaluates models on their ability to:
- Demonstrate deep subject matter expertise
- Apply complex reasoning to challenging problems
- Show consistent performance across varied domains
Methodology
All reported results use the 5-shot Chain-of-Thought prompting method, which included 5 examples per category in the prompt, as well as encouraging the models to think step by step. Few-shot CoT prompting is the approach used in the original paper. Here is the exact prompt:
The following are multiple-choice questions (with answers) about biology. Think step by step and then finish your answer with "The answer is (X)" where X is the correct letter choice.
Question: Which of the following represents an accurate statement concerning arthropods?
Options: A. They possess an exoskeleton composed primarily of peptidoglycan., B. They possess an open circulatory system with a dorsal heart., C. They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources., D. They lack paired, jointed appendages.
Answer: Let's think step by step. Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B), as arthropods have an open circulatory system with a dorsal tubular heart. The answer is (B).
Questions and Answers
Which of the following is not a way to form recombinant DNA?
Options: A. Translation, B. Conjugation, C. Specialized transduction, D. Transformation
Answer: Let's think step by step. The introduction of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique in molecular biology and scientific research. There are multiple ways foreign DNA can be introduced into cells including transformation, transduction, conjugation, and transfection. In contrast, (A) is not a way to form DNA: during translation the ribosomes synthesize proteins from RNA. The answer is (A).Which of the following is not known to be involved in the control of cell division?
Options: A. Microtubules, B. Checkpoints, C. DNA polymerase, D. Centrosomes, E. Cyclins, F. Mitochondria, G. Protein kinases, H. Fibroblast cells
Answer: Let's think step by step. Normal cells move through the cell cycle in a regulated way. At the checkpoint stage, they use information about their own internal state and cues from the environment around them to decide whether to proceed with cell division. Cues like these act by changing the activity of core cell cycle regulators inside the cell. The most common regulators are cyclins and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division. The answer is (H).Which of the following would most likely provide examples of mitotic cell divisions?
Options: A - cross section of muscle tissue, B - longitudinal section of a shoot tip, C - longitudinal section of a leaf vein, D - cross section of a fruit, E - cross section of a leaf, F - longitudinal section of a petal, G - longitudinal section of a seed, H - cross section of an anther (site of pollen production in a flower)