SWE-bench

Overview

SWE-bench is a benchmark for evaluating Large Language Models (LLMs) in software engineering contexts, introduced by Jimenez et al. in their seminal paper "Can Language Models Resolve Real-World GitHub Issues?". The benchmark comprises 500 tasks, each executed within an isolated Docker container. These tasks represent real-world GitHub issues from various repositories. Models must generate a "patch" to resolve each issue. The success of a model’s solution is determined by running unit tests against the generated patch.

A notable complexity of SWE-bench lies in its dual evaluation of both the agentic harness and the underlying foundation model. This leads to different methodologies adopted by foundation model labs when they report their results. Additionally, the benchmark’s computational requirements make it resource-intensive to reproduce results.

Methodology

To enable fair and consistent comparisons across foundation models, we use a minimal bash-tool-only agent harness, mini-swe-agent. Models are given a single tool — bash — and a system prompt describing the task. They must use standard command-line tools to navigate the codebase, identify the relevant code, and produce a patch.

We use the SWE-bench Verified subset of the dataset, which is a human-validated section released by OpenAI in August 2024. Each task in the split has been carefully reviewed and validated by human experts, resulting in a curated set of 500 high-quality test cases from the original benchmark. You can find more information about the Verified split of the dataset here.

All experiments are run on isolated cloud sandboxes. Latency is calculated starting from the first step the model takes within each task.

Results

Rankings by Performance

  1. Claude Opus 4.7

    • Accuracy: 82.00% ± 1.72
    • Cost/Test: $2.42
    • Latency: 441.99s

  2. Gemini 3.1 Pro Preview (02/26)

    • Accuracy: 78.80% ± 1.83
    • Cost/Test: $0.78
    • Latency: 312.26s

  3. Claude Opus 4.6 (Thinking)

    • Accuracy: 78.20% ± 1.85
    • Cost/Test: $1.22
    • Latency: 350.76s

  4. GPT 5.4

    • Accuracy: 78.20% ± 1.85
    • Cost/Test: $0.80
    • Latency: 307.12s

  5. GPT 5.3 Codex

    • Accuracy: 78.00% ± 1.85
    • Cost/Test: $0.46
    • Latency: 246.53s

Key Takeaways

  • The closing performance ranking shows that models proficient in bash and standard command-line tools for code navigation and editing perform well on SWE-bench.
  • Closed-source models generally outperform open-source models, especially on tasks that take between 15 minutes and 1 hour to complete.

Model Performance Summary

  • Claude Opus 4.7: Best performing with highest accuracy.
  • Performance variance based on model type (closed vs. open source) observed across various tasks.

Tool Use

All models are given a single tool: bash. Models must use standard command-line tools (grep, find, sed, etc.) to navigate codebases, search for relevant files, and apply edits. This evaluation structure aims to reflect each model's command-line fluency and problem-solving strategies.