← All papers

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez, Yang, Wettig et al. · 2024 · ICLR 2024

EvaluationRead on arXiv

Benchmarks LLMs on real GitHub issues from popular Python repos, requiring multi-file patches that pass the project's own test suite. SWE-bench has become the defining evaluation for coding agents.

Key Idea

SWE-bench evaluates whether an LLM (or an agent built on top of one) can resolve real software engineering tasks end-to-end: given a GitHub issue and the repository at the commit just before the fix, the model must produce a code patch that, when applied, makes the project's hidden test suite pass.

Dataset Construction

  • 2,294 task instances mined from 12 popular Python repos (django, sympy, scikit-learn, matplotlib, flask, requests, pytest, pylint, sphinx, astropy, seaborn, xarray)
  • Each instance includes: the issue text, the pre-fix codebase, and PASS_TO_PASS + FAIL_TO_PASS tests that gate correctness
  • Tasks are non-trivial: median fix touches multiple files, with realistic dependencies on existing project code
  • SWE-bench Verified (2024): a 500-instance subset hand-validated by ~90 professional software engineers at OpenAI, with ambiguous specs and broken tests removed, now the preferred reporting split

Why It's Hard

  • Patches must compile, integrate cleanly, and pass both the issue's regression tests and the existing test suite (no breaking other things)
  • Models must navigate a large repo, not just complete a snippet, typical context far exceeds a single file
  • Issues are written by humans, often underspecified or ambiguous, requiring the model to read code to infer intent

Results Trajectory

  • Original paper (2023): best system <2% resolved, frontier LLMs basically couldn't do it
  • 2024 with agent scaffolds (SWE-agent, Aider, Devin, AutoCodeRover): jumped to ~20 to 30% on the full set
  • 2025 on SWE-bench Verified: top systems (Claude, GPT, open agents) report 50%+, with the SOTA continuing to climb past 60%
  • Driven a whole ecosystem of coding agents and tool-use scaffolds (terminal access, file search, test runners)

Why It Matters

  • The first widely-adopted benchmark that measures agentic, multi-file, test-verified coding rather than isolated function completion (HumanEval, MBPP)
  • Verified split is now the reference number cited by Anthropic, OpenAI, Google, Meta in coding-model releases
  • Created a clear capability axis ("can it close real issues?") that maps to commercial value (Cursor, Cognition/Devin, Claude Code, Copilot Workspace all target this)

Key Takeaways for Interviews

  • SWE-bench tests end-to-end issue resolution with the project's own tests as the oracle, much stricter than snippet benchmarks
  • Always cite SWE-bench Verified (500) for current SOTA; the original 2,294 set has known noise (broken tests, underspecified issues)
  • Scores depend heavily on the agent scaffold around the model (file navigation, retrieval, test execution loop), not just the base LLM, a strong scaffold can double a model's score
  • Complementary to HumanEval/MBPP (snippets) and LiveCodeBench (contamination-resistant competitive programming), different signals
  • When designing a coding-agent system, SWE-bench is the standard way to regression-test the whole stack (model + retrieval + tool use + patch application)