<!-- LLM_VERSION_INFO
FORMAT: text/markdown
CONTENT_TYPE: article
ORIGINAL_URL: https://www.vals.ai/benchmarks/mmmu
ALTERNATE_VERSION: benchmarks/mmmu/index.html (text/html)
EXTRACTION_DATE: 2026-04-17T00:42:46.995Z

This is the markdown version with text-only content (images converted to alt-text).
For rich formatting with images, request the HTML version at: benchmarks/mmmu/index.html
-->

# MMMU Pro

## Multimodal Multi-task Benchmark

### Updated: 4/16/2026

### Task type: Overall

### Systems (66)

| Model | Accuracy | Cost In/Out | Latency |
|-------|----------|--------------|---------|
| [Gemini 3.1 Pro Preview (02/26)](/content/models/google_gemini-3.1-pro-preview/index.html) | 88.21% ± 0.78 | $2 / $12 | 76.99s |
| [Gemini 3 Flash (12/25)](/content/models/google_gemini-3-flash-preview/index.html) | 87.63% ± 0.79 | $0.5 / $3 | 27.86s |
| [Gemini 3 Pro (11/25)](/content/models/google_gemini-3-pro-preview/index.html) | 87.51% ± 0.79 | $2 / $12 | 48.10s |
| [GPT 5.4](/content/models/openai_gpt-5.4-2026-03-05/index.html) | 87.51% ± 0.79 | $2.5 / $15 | 65.57s |
| [Muse Spark](/content/models/meta_muse_spark/index.html) | 87.40% ± 0.80 | N/A | 48.27s |
| [GPT 5.2](/content/models/openai_gpt-5.2-2025-12-11/index.html) | 86.67% ± 0.82 | $1.75 / $14 | 96.32s |
| [Claude Opus 4.7](/content/models/anthropic_claude-opus-4-7/index.html) | 85.55% ± 0.85 | $5 / $25 | 70.80s |
| [Kimi K2.5](/content/models/kimi_kimi-k2.5-thinking/index.html) | 84.34% ± 0.87 | $0.6 / $3 | 110.01s |
| [Qwen 3.6 Plus](/content/models/alibaba_qwen3.6-plus/index.html) | 84.16% ± 0.88 | $0.5 / $3 | 111.94s |
| [Claude Opus 4.6 (Thinking)](/content/models/anthropic_claude-opus-4-6-thinking/index.html) | 83.87% ± 0.88 | $5 / $25 | 119.01s |

### Key Takeaways

- [Gemini 3.1 Pro Preview (02/26)](/content/models/google_gemini-3.1-pro-preview/index.html) leads with 88.21%, narrowly edging out [Gemini 3 Flash (12/25)](/content/models/google_gemini-3-flash-preview/index.html) (87.63%) and [Gemini 3 Pro (11/25)](/content/models/google_gemini-3-pro-preview/index.html) (87.51%).
- Models are now approaching the performance of the best human experts (88.6%), with the top model only 0.4 percentage points away. Progress on this benchmark seems to be approaching an asymptotic limit.
- Most of the top models are reasoning models. Generally, the reasoning models take noticeably longer than their non-reasoning counterparts.

## Results

According to the [original MMMU research](https://mmmu-benchmark.github.io/), human expert performance ranges from 76.2% for the worst-performing experts to 88.6% for the best-performing experts. Our evaluation shows that the leading AI models have now surpassed the lower bound and are approaching the upper bound of human expert performance.

### MMMU Pro

- Input Cost
- Accuracy
- Alibaba
- Anthropic
- Google
- Meta
- Mistral
- Moonshot AI
- OpenAI
- xAI

### Dataset and Context

The Multimodal Multi-task Benchmark (MMMU) follows a similar methodology to its predecessor, MMLU, but the multiple-choice questions asked include both text and images. MMMU Pro encompasses over 1,000 high-quality tasks spanning 30 subjects in 6 major disciplines:
- Arts & Design
- Business
- Science
- Health & Medicine
- Humanities & Social Sciences
- Tech & Engineering

We based this benchmark on the standard 4-option multiple-choice format containing approximately 1,700 questions from the [official Hugging Face dataset](https://huggingface.co/datasets/MMMU/MMMU_Pro). The benchmark focuses specifically on how well models can process and reason about problems where images are interleaved with text, requiring sophisticated visual understanding and cross-modal reasoning capabilities.

MMMU Pro is particularly valuable because it tests the models’ abilities to solve graduate-level questions where visual information is critical to finding the correct answer.

### Methodology

We adhered closely to the official MMMU Pro evaluation protocol with the following implementation details:

1. **Prompt Structure**: We used the chain-of-thought prompt from the [original MMMU repository](https://github.com/MMMU-Benchmark/MMMU/blob/main/mmmu-pro/prompts.yaml). Each question-answer set followed this format:
   
   ```
   Which of the following best explains the overall trend shown in the <image>?
   A. Migrations to areas of Central Asia for resettlement
   B. The spread of pathogens across the Silk Road
   C. Invasions by Mongol tribes
   D. Large-scale famine due to crop failures
   Answer the preceding multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' (without quotes) where LETTER is one of options. Think step by step before answering.
   ```

2. **Image Processing**: All `<image>` tags were replaced with actual image bytes during model inference, replicating the original methodology exactly.

3. **Standardization**: We established a consistent baseline by using a default configuration of 8192 maximum output tokens for all models to ensure that outputs were not truncated. All models were run with a temperature of 0.

4. **Parsing Adaptation**: We modified the answer extraction regex to handle markdown output from some models, ensuring reliable parsing across all responses.

5. **Statistical Validity**: This was a pass@1 evaluation (one attempt per question), with the large dataset size (1,700 questions). We found a standard deviation of approximately 1% (calculated using the methodology from [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://arxiv.org/pdf/2411.00640)).

6. **Thinking Models Configuration**: For models with “thinking” capabilities (Claude 3.7 Sonnet Thinking, o1, etc.), we set the maximum output token limit to 16,384 to accommodate extended reasoning.