HF's Joël Niklaus Ships Harness Leaderboard: 28,849 Trials Across 9 Coding Agent Harnesses; Author Reports mini-swe-agent ~51.4% Average

Core Background & Industry Pain Points

Open coding models are often scored inside Claude Code, Codex, OpenCode, and similar harnesses, so published numbers mix model skill with scaffolding and teams cannot tell whether to swap the model or the scaffold. On 2026-11-05, Hugging Face Harness Optimization researcher Joël Niklaus (@joelniklaus) released the Harness Leaderboard and stated that each comparison changes only the harness while pinning model, provider, and reasoning effort, with three attempts per task.

Architecture Highlights & Internals

The board covers nine coding-agent harnesses (Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code, OpenClaw). Models so far: GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint. Benchmarks: Terminal-Bench 2.1 and DeepSWE. The Space README says runs use Hugging Face infrastructure (Inference Providers / Sandbox / Jobs) with public trajectories in a bucket. Looking up answers, copying transcripts, or special-casing the grader counts as a fail under the board's own rule.

Authoritative Benchmarks & Measured Scores

All figures below are author/leaderboard-reported, not independently reproduced here: 28,849 trials and about $4,746 of inference. Averaged across the three models, bash-only mini-swe-agent resolves about 51.4% of tasks versus openclaw at about 27.3%; four others are statistically tied with the leader. Pi is about 49.4% at roughly $0.29 per solved task, about 45% cheaper than mini-swe-agent's ~$0.53. Best harness is model-specific: mini-swe-agent on GLM-5.3-Flash, Hermes Agent on DeepSeek-V4.1-Flash, Kimi Code on Flint. OpenCode costs about $3.35 per solved task—about 11.7× Pi.

Developer Hands-on Guide

Start at https://huggingface.co/spaces/joelniklaus/harness-leaderboard and the public trajectory bucket, then use Harness Lens to compare prompts, tools, and compaction. Pick by the model you actually run—not the grand average alone—and weigh cost plus three-attempt reliability. This is an individual research leaderboard, not a Hugging Face company product launch.