Independent benchmarker Artificial Analysis announced on X on November 4 (17:34 UTC) a new version of its API performance benchmarking: the headline metric moves from output speed to Time per Task, the time to finish one Intelligence Index task (output tokens per task divided by end-to-end output speed). It covers about 1,000 serverless endpoints, measures speed from sending the request to the final token, and adds multi-turn agentic coding and knowledge-work workloads. Per its data, GPT-6 Astra at max effort runs at 51 tokens/s yet finishes a task in about 17 minutes, against about 26 for Claude Opus 5.5 (max) and about 29 for Gemini 4 Carbon (high).
Key Takeaways
- ✓Time per Task equals output tokens per Intelligence Index task divided by end-to-end output speed, so a model that is fast per token but verbose can finish later than a slower model that uses fewer tokens.
- ✓End-to-end output speed runs from sending the request to the final token, including the wait before the first token (prefill and hidden reasoning); output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer.
- ✓GPT-6 Astra (max) runs at 51 tokens/s with about 50k output tokens per task and takes about 17 minutes; Claude Opus 5.5 (max, 111 tokens/s) takes about 26 minutes and Gemini 4 Carbon (high, 71 tokens/s) about 29, using roughly 120k to 170k tokens per task.
- ✓At each lab's API default effort (medium for all three), GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 cut Time per Task by about 3 to 5x and Cost per Task by about 2 to 4x, scoring about 5 to 11 points lower; Opus 5.5 drops from about 26 minutes and $10.04 to about 6.8 minutes and $2.45.
- ✓The official methodology page lists this as version 3.0.0 (October 30), with runs now every 6 hours and a new endpoint error rate metric.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the Industry Pain Point
API performance comparisons have long centered on output tokens per second. As workloads become more agentic, what developers care about is how long a job takes to finish. Artificial Analysis posted on X that a model that is fast per token but verbose can take longer than a slower model that uses fewer tokens, so it made Time per Task its headline metric.
Architecture Highlights and Mechanism
According to the post and the official methodology page (version 3.0.0, October 30), there are four changes. Time per Task is output tokens per task divided by end-to-end output speed. Speed now runs from sending the request to the final token, counting prefill and hidden reasoning, with token counts taken from each provider's API. Multi-turn coding and knowledge-work agent sessions join the short, document-length and long-context prompts, averaged over the past 5 days. Key models are measured at both API default effort and highest effort.
Benchmarks and Measured Results
Per Artificial Analysis, GPT-6 Astra and GPT-6.1 Sol hold most of the Intelligence vs. Time per Task Pareto frontier from about 45 to 55 on the Intelligence Index. GPT-6 Astra (max) runs at only 51 tokens/s but uses about 50k tokens and about 17 minutes per task; Claude Opus 5.5 (max) runs at 111 tokens/s and takes about 26 minutes; Gemini 4 Carbon (high) runs at 71 tokens/s and takes about 29 minutes. Default medium effort cuts time by about 3 to 5x and cost by about 2 to 4x, at about 5 to 11 points lower on the index.
Getting Started for Developers
When choosing a model for agents, compare Time per Task and Cost per Task on Artificial Analysis rather than tokens per second alone, and start with each lab's default effort before turning it up. Artificial Analysis also notes that endpoints optimized only for decode speed, for example by queuing requests at lower batch sizes or holding output for one burst, drop the most under the new measure. All figures are Artificial Analysis's own measurements.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.