Monthly-SWEBench

The Rise and Fall of Static Benchmarks

SWE-bench (2023) took a practical approach to evaluating coding agents: derive tasks from real GitHub issues in popular Python repositories and use the existing test suites to judge correctness. The benchmark gained wide adoption, and model performance on it improved rapidly over the following years.

But scores went up faster than real-world usefulness.

As more teams optimized for SWE-bench, scores kept rising while practitioners reported little matching improvement in day-to-day coding assistance. The benchmark was being overfitted, not through explicit cheating, but through the gradual contamination of training data that inevitably absorbs widely-studied public benchmarks.

SWE-bench Verified attempted to address quality concerns. OpenAI partnered with the original authors to have human annotators review 500 tasks from the original 2,294, filtering out under-specified or ambiguous problems. This improved the benchmark’s reliability, but it was still a static, fixed set. OpenAI eventually announced they would no longer evaluate on SWE-bench Verified, citing concerns that even the curated subset was becoming saturated.

SWE-bench-Pro pushed further with harder tasks. Yet the underlying problem remains: any fixed set of tasks has an expiration date. The moment a benchmark is published, its clock starts ticking toward obsolescence.

SWE-bench Live introduced periodic updates with new tasks, taking a step toward liveness. However, it remains Python-focused, covers only bug fixes, and lacks rigorous instruction-test alignment verification, leaving room for ambiguous or under-specified tasks to slip through.

Why We Need a Living, Verified, and Broad Benchmark

The problems outlined above are not isolated issues. They reflect fundamental gaps in existing benchmarks: liveness (static task sets expire), verification (task quality is not systematically guaranteed), and breadth (limited to Python and bug fixes only). A next-generation software engineering benchmark must address all three. Monthly-SWEBench is built around six pillars spanning these dimensions.

The following table summarizes where each benchmark stands across these dimensions:

The Monthly-SWEBench Pipeline

Our benchmark is not hand-crafted. It is produced by a multi-stage pipeline that starts from the global stream of open-source activity and distills it into high-quality, verified tasks. Here is what happens at each stage.

Conclusion

The era of static, unverified software engineering benchmarks is ending. As language models grow more capable, three problems become critical: fixed task sets get overfitted, poorly constructed tasks give misleading signals, and narrow scope fails to reflect the diversity of real-world engineering. A benchmark must address all three.

Monthly-SWEBench addresses all three:

• Live: monthly releases of ~100 tasks from freshly merged PRs across fresh repositories ensure models cannot memorize their way to high scores
• Verified: every task passes strict instruction-test alignment review, behavioral test quality checks, fail-to-pass dual-phase verification, and solver-perspective auditing to ensure no hidden requirements
• Broad: covering Python, Go, C++, Java, Rust, TypeScript, JavaScript and more across ~20 repositories, with both bug fixes and feature additions covering a fuller spectrum of real-world software engineering than any prior benchmark

Honest evaluation is a prerequisite for genuine progress. We need benchmarks that reflect where AI coding agents actually are, not where they were six months ago.

@misc{monthlyswebench2026,
    title={Monthly-SWEBench: A Living, Rigorously Verified Benchmark
           for Real-World Software Engineering},
    author={Haiyang Shen and Xinbo Xu and Xuanzhong Chen and Wendong Xu and Elvis Zhang and Kaiyuan Chen and Xiaobo Hu and Rui Wang and Yang Liu and Yixin Ren and Yuan Gong and Liang Chen and Kuan Li},
    year={2026},
    note={Blog post: https://unipat.ai/benchmarks/MonthlySWEBench}
}