<!-- LLM_VERSION_INFO
FORMAT: text/markdown
CONTENT_TYPE: article
ORIGINAL_URL: https://www.vals.ai/benchmarks/mmlu_pro
ALTERNATE_VERSION: benchmarks/mmlu_pro/index.html (text/html)
EXTRACTION_DATE: 2026-04-17T00:45:41.070Z

This is the markdown version with text-only content (images converted to alt-text).
For rich formatting with images, request the HTML version at: benchmarks/mmlu_pro/index.html
-->

# MMLU Pro

Academic multiple-choice benchmark covering 14 subjects including STEM, humanities, and social sciences.

## Systems

| Model   | Accuracy        | Cost In / Out | Latency  |
|---------|-----------------|----------------|----------|
|  Gemini 3.1 Pro Preview (02/26)  | 90.99%± 0.28     | $2 / $12       | 23.91s   |
|  Gemini 3 Pro (11/25)            | 90.10%± 0.30     | $2 / $12       | 24.27s   |
|  Claude Opus 4.7         | 89.87%± 0.31     | $5 / $25       | 187.83s  |
|  Claude Opus 4.6 (Thinking) | 89.11%± 0.45     | $5 / $25       | 49.45s   |
|  Gemini 3 Flash (12/25)         | 88.59%± 0.32     | $0.5 / $3      | 36.25s   |
|  Claude Opus 4.1 (Thinking) | 87.92%± 0.33     | $15 / $75      | 27.67s   |
|  Qwen 3.6 Plus               | 87.67%± 0.32     | $0.5 / $3      | 46.18s   |
|  GPT 5.4                   | 87.48%± 0.42     | $2.5 / $15     | 28.88s   |
|  Claude Sonnet 4.5 (Thinking) | 87.36%± 0.39     | $3 / $15       | 38.20s   |
|  Claude Sonnet 4.6         | 87.34%± 0.43     | $3 / $15       | 55.41s   |

## Key Takeaways

- [Gemini 3.1 Pro Preview (02/26)](/content/models/google_gemini-3.1-pro-preview/index.html) leads the benchmark with 90.99% accuracy. [Gemini 3 Pro (11/25)](/content/models/google_gemini-3-pro-preview/index.html) follows in second place at 90.10%, followed by [Claude Opus 4.7](/content/models/anthropic_claude-opus-4-7/index.html) at 89.87%.
- It’s clear that this benchmark is largely saturated - there will be diminishing utility to model providers continuing to hill climb on MMLU Pro.

## Dataset and Context

[MMLU](https://arxiv.org/pdf/2009.03300) is a commonly-used academic evaluation that tests models on multiple-choice questions on subjects like physics, chemistry, etc. [MMLU Pro](https://arxiv.org/abs/2406.01574) is an improved version of MMLU, focusing on data quality and diversity. MMLU Pro consists of over 12,000 question-answer pairs, created through:

- Filtering and enhancing classic MMLU questions
- Expanding answer options from 4 to 10 choices
- Incorporating new sources (STEM Website, TheoremQA, SciBench)
- Expert verification of questions

The benchmark evaluates models on their ability to:

- Demonstrate deep subject matter expertise
- Apply complex reasoning to challenging problems
- Show consistent performance across varied domains

### Methodology

All reported results use the 5-shot Chain-of-Thought prompting method, which included 5 examples per category in the prompt, as well as encouraging the models to think step by step. Few-shot CoT prompting is the approach used in the original paper. Here is the exact prompt:

```markdown
The following are multiple-choice questions (with answers) about biology. Think step by step and then finish your answer with "The answer is (X)" where X is the correct letter choice.
Question: Which of the following represents an accurate statement concerning arthropods?
Options: A. They possess an exoskeleton composed primarily of peptidoglycan., B. They possess an open circulatory system with a dorsal heart., C. They are members of a biologically unsuccessful phylum incapable of exploiting diverse habitats and nutrition sources., D. They lack paired, jointed appendages.
Answer: Let's think step by step. Peptidoglycan is known to comprise the plasma membrane of most bacteria, rather than the exoskeleton of arthropods, which is made of chitin, which rules out (A). The answer (C) is false because arthropods are a highly successful phylum. Likewise, arthropods have paired, jointed appendages, which rules out (D). The only remaining option is (B), as arthropods have an open circulatory system with a dorsal tubular heart. The answer is (B).
```

### Questions and Answers

1. **Which of the following is not a way to form recombinant DNA?**  
   **Options:** A. Translation, B. Conjugation, C. Specialized transduction, D. Transformation  
   **Answer:** Let's think step by step. The introduction of foreign DNA or RNA into bacteria or eukaryotic cells is a common technique in molecular biology and scientific research. There are multiple ways foreign DNA can be introduced into cells including transformation, transduction, conjugation, and transfection. In contrast, (A) is not a way to form DNA: during translation the ribosomes synthesize proteins from RNA. The answer is (A).

2. **Which of the following is not known to be involved in the control of cell division?**  
   **Options:** A. Microtubules, B. Checkpoints, C. DNA polymerase, D. Centrosomes, E. Cyclins, F. Mitochondria, G. Protein kinases, H. Fibroblast cells  
   **Answer:** Let's think step by step. Normal cells move through the cell cycle in a regulated way. At the checkpoint stage, they use information about their own internal state and cues from the environment around them to decide whether to proceed with cell division. Cues like these act by changing the activity of core cell cycle regulators inside the cell. The most common regulators are cyclins and cyclin-dependent kinases. Fibroblast cells do not play any role in cell division. The answer is (H).

3. **Which of the following would most likely provide examples of mitotic cell divisions?**  
   **Options:** A - cross section of muscle tissue, B - longitudinal section of a shoot tip, C - longitudinal section of a leaf vein, D - cross section of a fruit, E - cross section of a leaf, F - longitudinal section of a petal, G - longitudinal section of a seed, H - cross section of an anther (site of pollen production in a flower)