Agent Error Dataset (AED) Released: 50,000+ Scaled Error-Diagnosis Pairs Power Failure Analysis and Post-Training

Background and the Problem

Traditional LLM agent training predominantly relies on sparse binary rewards, discarding unsuccessful rollouts that contain rich diagnostic information regarding state transitions, faulty decisions, and stack traces. Lacking structured failure data, current agents struggle with localized self-correction and recurrent pitfalls across complex multi-turn trajectories.

Architecture and How It Works

UIUC researchers introduced the Agent Error Dataset (AED), the largest-scale failure analysis dataset comprising 50,228 error-diagnosis pairs across 9,961 source tasks, 33 environments, 19 harness families, and 23 policy models. AET (Agentic Error-to-Training) powers the data collection across 5 stages: natural failure harvesting, causal diagnosis and repair generation, recorded evidence validation, controlled checkpoint replay, and multi-view training derivation (diagnosis vs. recovery).

Benchmarks and Measured Results

Empirical validation across controlled replay and post-training demonstrates massive improvements:

1. Pass Rate Spikes by 32.7%: Across 3,062 matched replay pairs, first-proposal corrections increase verifier pass rates from 18.4% to 51.1% (+32.7 percentage points).
2. Exact-Step Failure Diagnosis at 63.6%: Fine-tuning Qwen3-8B on AED's full-diagnosis slice elevates exact-step diagnosis agreement with teacher labels from 47.2% to 63.6% on a 943-case holdout, outperforming prompted frontier baselines (54.7%).
3. Repair Training Outperforms Success-Only: In WebShop-lite policy training, action-level repair training achieves 6.67 percentage points higher score than conventional success-only training.

Getting Started for Developers

AED provides ready-to-train JSONL datasets formatted for SFT and DPO post-training. Practitioners training autonomous coding agents can leverage AED to train dedicated runtime diagnostic critics or fine-tune core policies with innate error recovery behaviors, markedly improving agent resilience in production environments.