CaseLaw (v2)
Overview
Proprietary benchmark crafted with industry experts on non-public datasets. Updated: 4/16/2026.
Private question-answer benchmark over Canadian court cases.
Task Type
Overall
Systems
| Model | Accuracy | Cost In/Out | Latency |
|---|---|---|---|
| GPT 5.1 | 73.42%± 0.75 | $1.25 / $10 | 23.59s |
| GPT 4.1 | 69.88%± 0.28 | $2 / $8 | 6.66s |
| GPT 5 Mini | 68.49%± 1.13 | $0.25 / $2 | 42.51s |
| Claude Opus 4.7 | 68.38%± 0.43 | $5 / $25 | 17.68s |
| GPT 5 | 66.45%± 0.65 | $1.25 / $10 | 123.42s |
| GPT 5.2 | 66.02%± 0.57 | $1.75 / $14 | 86.23s |
| Grok 4 | 65.81%± 0.21 | $3 / $15 | 32.34s |
| Grok 4 Fast (Reasoning) | 65.70%± 0.43 | $0.2 / $0.5 | 23.71s |
| Kimi K2 Thinking | 65.70%± 1.82 | $0.6 / $2.5 | 91.05s |
| Gemini 3.1 Pro Preview (02/26) | 64.84%± 1.24 | $2 / $12 | 56.78s |
Key Takeaways
- GPT 5.1 emerged as a strong first-place performer with 73.42% accuracy, demonstrating excellent legal reasoning capabilities with faster processing times.
- A common issue for the models was identifying only parts of relevant document sections, relying more on their general knowledge rather than specific document context.
- While there is significant room for improvement, the top-performing models demonstrated high accuracy, making them well-suited for these tasks.
Context
The CaseLaw v2 benchmark report presents a study of using LLMs for litigation, particularly how law relates to public court systems. Case law in the US and Canada is challenging to use at scale due to strict licensing requirements preventing LLMs from being trained on them. Organizations that provide access to substantial case histories do so only for legal use at significant cost.
Testing models on this data offers several benefits:
- Practicing lawyers will query models using references to recent cases that foundational models or applications have not been exposed to during their training.
- Most legal LLM evaluations have focused solely on US law; our study expands to explore their application in international legal systems.
In collaboration with the legaltech startup Jurisage, we announce the creation of our latest dataset, "CaseLaw v2." Building on our previous benchmark, we curated harder, more up-to-date questions, providing an enhanced evaluation of large language models’ capabilities in legal document analysis and case law reasoning. Models were performing quite well (scoring almost 90%) on our CaseLaw benchmark; however, with CaseLaw v2, no model obtains above 80% accuracy.
This dataset remains private with cases from recent court decisions beyond models’ training cutoff dates. The evaluation tasks test real-world legal reasoning capabilities based on feedback from legal practitioners.
Benchmark Tests
The benchmark tests models along seven dimensions, including:
- Retrieving the most important cases for a given query.
- Answering questions over multiple documents.
- Providing multi-point answers that have several components.
- Performing calculations to arrive at an answer.
- Reading over tables.
- Working chronologically over some input data.
- Understanding terms of art relevant to case law research.
The benchmark contains tests requiring answers based on a single relevant case as well as more complex questions that refer to multiple cases. In total, it contains 300 tests in our validation split and 104 tests in our test split.
Legal document analysis presents unique challenges due to complex legal language, precise interpretation requirements, and the need to identify relevant precedents—ideal for testing advanced language models. Each evaluation presents models with a substantial amount of legal information, requiring sophisticated analysis. Questions test both extractive capabilities (finding specific information) and reasoning abilities (understanding legal implications and relationships).
Example Questions
- Example Question 1: From these X cases, which one(s) are most relevant as a precedent for Y situation?
- Example Question 2: From the cases provided, what is meant by X? Answer the question only with the relevant excerpt or multiple excerpts from the documents, and do not include additional description or explanation.
Performance Analysis
The performance spread highlights the complexity of legal reasoning tasks, with frontier models clearly outperforming alternatives by up to ~25%. The accuracy vs. efficiency analysis shows varied approaches to legal reasoning. GPT 5.1 provides the highest accuracy, while GPT 4.1 offers an excellent balance of performance and speed, and GPT 5 Mini delivers strong results at the lowest cost.
Changelog
- This benchmark was updated 11/17/25 to upgrade the evaluator model from GPT 4 Turbo to GPT 5.1.
- This benchmark was updated 02/26/26 to reduce error.