AI & Agents
SWE-sweep
Last updated 2026-10-01 · benchmark measured 2026-11-10 — deterministic & reproducible
A benchmark measuring how many bugs AI agents can find and fix in real codebases.
SWE-sweep at a glance
SWE-sweep is a benchmark that tests how well AI coding agents can find and fix multiple real bugs in a codebase without being told what or where the bugs are, for researchers and teams evaluating language model agents.
Good for
- Evaluating an LLM agent's ability to discover bugs on its own, with no hints about type or location
- Comparing different coding agents and models on resolved bug rate, cost, turns, and tokens used
- Testing whether an agent's fixes introduce regressions or break existing tests
Questions people ask
What does SWE-sweep actually measure?
It measures how many bugs a coding agent can find and fix on its own in a real repository that contains many bugs at once, with no information given about what the bugs are or where they are located.
How are the test repositories and bugs chosen?
Tasks are built from real issue-pull request pairs in open-source repositories, picking a single commit where the maximum number of bugs exist at the same time, and filtering to keep only bugs that can be discovered just by reading the repository.
How is an agent's fix judged as successful?
A bug counts as resolved only if hidden tests tied to that bug all pass after the fix and the repository's original test suite still passes, which also checks for regressions.
Does the agent get any hints about the bugs?
No, the agent only receives the repository at a fixed starting commit and an instruction to find and fix as many bugs as possible, with no issue descriptions, filenames, or line ranges.
Is there a way to compare different AI models and agents on this benchmark?
Yes, SWE-sweep includes a public leaderboard that compares models and agents by percentage of bugs resolved, total and per-repository cost, turns, and tokens used.
Written by Legit.Show from swesweep.com on 2026-10-26. It does not change the score.
Are you the maker of SWE-sweep? Claim this listing. It is free, takes a meta tag, a DNS record or GitHub admin rights, and never changes the score.
Launched something? Add your product, free.
To cite this score: legit.show/s/swesweep-com/2026-11-10. That address never changes; this page moves with every re-measure.
Legit.Show scored SWE-sweep 70/100 on 2026-11-10, measured across all 7 frames from its public surface. legit.show/s/swesweep-com/2026-11-10
Is SWE-sweep production-ready?
Legit.Show scores SWE-sweep 70 out of 100 — the simple average of its 7 measured frames. Legit.Show ran its deterministic 7-Frame production-readiness benchmark on SWE-sweep (public-surface assessment), measured from the public surface with no LLM in the scoring path. Its strongest frame is Reliability; its weakest is Privacy. Every frame it averages is published with its evidence on the Legit.Show listing.
The 7 Frames
- Performance — 73/100
- Accessibility — 95/100
- Security — 45/100How this compares across the catalogue: The Web Security Baseline
- Privacy — 25/100How this compares across the catalogue: The Privacy Gap
- Reliability — 100/100
- Standards — 92/100
- Discoverability — 60/100
What we measured
- Security headers present: HSTS.
- No Content-Security-Policy.
- Served over HTTPS with a valid certificate.
- Real Lighthouse performance run — 218 ms to first byte.
- Returns a proper 404 for unknown routes.
- 3 of 3 sampled routes reachable.
- No privacy policy found.
- Sets cookies / loads scripts with no consent prompt.
Who it's for
AI researchers · LLM developers · Software engineering teams · Benchmark creators
Sources and updates
- Description
- Taken from swesweep.com's own website on 2026-10-01.
- Operator
- Meta Platforms, Inc, as named in swesweep.com's terms, privacy page or footer.
- Benchmark
- Measured by Legit.Show from the live site, the way any visitor sees it — with no access to its code or accounts. Same method for every product, and no AI decides the score. Last checked 2026-11-10.
Put together from public information, without SWE-sweep's involvement. If anything here is wrong, tell us and a person will check it.
Visit SWE-sweep → · Alternatives to SWE-sweep → · How this was measured →