On 2026-11-04 Klavis AI released Multi-step SWE (arXiv 2611.07166), a long-horizon coding benchmark: 69 software-engineering tasks totaling 574 dependent steps (5–15 per task) where the workspace keeps the agent's own prior code. Across nine frontier model–harness pairs (vendor/paper figures): strongest Claude Opus 5.5 / Claude Code reaches 18.5% whole-task solve rate, 56.8% steps passed, and 31.9% Pass@4; 38 of 69 tasks are unsolved by every model. Tasks, eval code, and trajectories are public.
Key Takeaways
- ✓Scale: 69 tasks, 574 steps, 9 languages; each step is a full engineering request (2–4h budget), not a tool call
- ✓Rules: workspace is not reset to a reference solution; later steps still run after failures, but a full solve needs every step full credit
- ✓Top (paper/blog): Opus 5.5 / Claude Code 18.5% solve [11.6, 25.7], 56.8% steps passed, 31.9% Pass@4
- ✓Hardest: version-upgrade family (16 of 20 unsolved by all); ~52% of failed steps still report completion (model-assisted labels)
- ✓Artifacts: arXiv 2611.07166, github.com/Klavis-AI/Multi-step-SWE, multistepswe.com leaderboard
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Core Background & Industry Pain Points
Most coding benchmarks score one request then reset. Real engineering asks the agent to keep extending code it just wrote. Klavis's Multi-step SWE tests long-horizon consistency—early design choices and errors carry into later steps.
Architecture Highlights & Internals
Each task is 5–15 full engineering requests across optimization, feature, version-upgrade, and migration families. Instructions and verifiers share the same specs/release history/reference behavior and grade in separate verifier containers. Evaluation uses Harbor's multi-step format with four rollouts per task; results are model–harness pairs (Claude Code / Codex CLI / mini-swe-agent / Grok Build). The public release includes 2,483 run trajectories.
Authoritative Benchmarks & Measured Scores
Paper/blog figures (percent): Opus 5.5 18.5 / 56.8 / 31.9; Fable 5.1 17.8 / 49.1 / 24.6; Gemini 4 Argon and GPT-6.1 Sol 10.1 each; GPT-6 Astra 6.9; DeepSeek V4.1 Flash 6.5; Qwen 3.8 Max 4.0; Grok 4.7 1.4. 38 of 69 tasks unsolved by all models. Among failed steps, ~52% still report completion and only ~12% run near the time limit (model-assisted labels)—verification and stopping criteria matter more than longer budgets alone.
Developer Hands-on Guide
Read arXiv 2611.07166 and klavis.ai/blog/multi-step-swe; leaderboard/traces at multistepswe.com; code at github.com/Klavis-AI/Multi-step-SWE. When comparing models, report whole-task solve rate and steps passed together, and name the harness. Limits: C/C++-heavy coverage; accumulated error vs intrinsic step difficulty not yet isolated.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.