A Stanford/Cornell team (E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Ré) posted arXiv 2611.08417 (2026-11-05) on "benchmark maintenance debt": even Terminal-Bench 2.0, which had human and LM red-teaming before release, still had defects in about 30% of tasks (28/89)—instruction–verifier mismatches, external-state drift, or overly tight resource budgets. Repairs ship as open Terminal-Bench 2.1. Re-running 19 agents moved individual scores by up to ~12 points and ranks by up to three places, while overall order stayed highly correlated (Spearman ρ=0.958). Extending timeouts largely closed a ~20 pp harness gap on GPT 5.4.
Key Takeaways
- ✓Defect rate: 28/89 tasks repaired; ~57% instruction–verifier mismatch, ~43% task drift, ~29% environment/resource issues (overlapping)
- ✓Impact: across 19 agents, individual scores move up to ~12 points and ranks up to 3 places; Spearman ρ=0.958
- ✓Example fix: query-optimize verifier required byte-identical DB files the prompt never stated—constraint made explicit; five tasks rose from ≤15% to >50% pass
- ✓Resources: >50% of failed trials at default budget are timeouts; extending to 32× largely closes a ~20 pp Codex vs Terminus-2 gap on GPT 5.4
- ✓Artifacts: github.com/harbor-framework/terminal-bench-2-1 and PR #53; workflow folded into Terminal-Bench ongoing review
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and Pain Point
Coding-agent leaderboards shape product claims and buying decisions, but executable benchmarks are not static exams: models find valid solutions verifiers reject, external packages and APIs drift, and tight CPU/memory/timeouts can fail the oracle itself. The authors call the backlog that decouples scores from intended capability "benchmark maintenance debt."
Architecture and Mechanism
The paper describes a continuous-validation pipeline: evidence collection (task files, traces, community reports) → an auditor drafts labeled repairs → /oracle rebuilds and runs the reference solution → maintainers gate the merge. Defects fall into instruction–verifier mismatch, task drift, and environment/resource issues. Resource repairs leave at least 1.5× slack for the oracle/verifier. Managed auditors Claude Code (Opus 4.6) and Codex (GPT-5.4) agree on primary labels at κ=0.80.
Benchmarks and Results
Re-evaluating 19 agents on Terminal-Bench 2.0 vs 2.1 yields Spearman ρ=0.958 and an OLS fit near TB2.1≈2.1+1.04·TB2.0. Gains are uneven—e.g., Claude Code + Opus 4.6 sits above the fit and climbs three ranks. At default budgets, timeouts can be ~68% of failures under the claude-code harness. On a 34-task Harbor-Index v0.1 sample the workflow hit ~86% precision (12/14 confirmed), so it transfers but is not perfect.
Developer Guide
When using Terminal-Bench for model choice, record the version (2.0 vs 2.1), harness, and time/memory budgets—do not quote one scalar. Artifacts: harbor-framework/terminal-bench-2-1, with change discussion in terminal-bench-2 PR #53. Community defects can use the /audit and /oracle Actions, with human merge gates.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.