AICoder

AI Coding Live News

First-hand AI coding updates from 𝕏/Twitter frontier labs and researchers, distilled by AI with entity benchmarks.

Explore live on 𝕏 ↗

AI Coding Live News

A
Artificial AnalysisArtificialAnlys·
👀16
🔥82
View on 𝕏 ↗

Artificial Analysis Makes Time per Task Its Headline Performance Metric: GPT-6 Astra Finishes a Max-Effort Task in About 17 Minutes, Ahead of Opus 5.5 and Gemini 4 Carbon

Independent benchmarker Artificial Analysis announced on X on November 4 (17:34 UTC) a new version of its API performance benchmarking: the headline metric moves from output speed to Time per Task, the time to finish one Intelligence Index task (output tokens per task divided by end-to-end output speed). It covers about 1,000 serverless endpoints, measures speed from sending the request to the final token, and adds multi-turn agentic coding and knowledge-work workloads. Per its data, GPT-6 Astra at max effort runs at 51 tokens/s yet finishes a task in about 17 minutes, against about 26 for Claude Opus 5.5 (max) and about 29 for Gemini 4 Carbon (high).

Key Takeaways

  • Time per Task equals output tokens per Intelligence Index task divided by end-to-end output speed, so a model that is fast per token but verbose can finish later than a slower model that uses fewer tokens.
  • End-to-end output speed runs from sending the request to the final token, including the wait before the first token (prefill and hidden reasoning); output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer.
  • GPT-6 Astra (max) runs at 51 tokens/s with about 50k output tokens per task and takes about 17 minutes; Claude Opus 5.5 (max, 111 tokens/s) takes about 26 minutes and Gemini 4 Carbon (high, 71 tokens/s) about 29, using roughly 120k to 170k tokens per task.
  • At each lab's API default effort (medium for all three), GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 cut Time per Task by about 3 to 5x and Cost per Task by about 2 to 4x, scoring about 5 to 11 points lower; Opus 5.5 drops from about 26 minutes and $10.04 to about 6.8 minutes and $2.45.
  • The official methodology page lists this as version 3.0.0 (October 30), with runs now every 6 hours and a new endpoint error rate metric.
Read full story
Linked Models & Agents:
⚡ GPT-6 Astra⚡ GPT-6.1 Sol⚡ Claude Opus 5.5⚡ Gemini 4 Carbon🌐 Artificial Analysis🌐 OpenAI🌐 Anthropic🌐 Google
#Artificial Analysis#Time per Task#Benchmark#API performance#Agentic#Latency
S
SemiAnalysis@SemiAnalysis_·
👀94
🔥85
View on 𝕏 ↗

SemiAnalysis releases GDPx, an agentic benchmark of 206 tasks across 11 high-value domains where the top model, Claude Opus 5.5, scores just 12%

On Nov 2, semiconductor and AI research firm SemiAnalysis released GDPx, a benchmark of agentic capability across 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement (RSI), hardware design, materials science, alignment and software engineering. Its 206 tasks were built by 14 specialist data partners, with SemiAnalysis doing final QA. On the official leaderboard Claude Opus 5.5 leads at 12.0%, followed by Gemini 4 Carbon at 10.9% and GPT-6 Astra at 9.6%; nearly every model scores 0 on finance and legal. SemiAnalysis says safety classifiers drag down Opus 5.5 and GPT-6 Astra, and plans a new version every 3–6 months. All scores are as reported by SemiAnalysis.

SemiAnalysis releases GDPx, an agentic benchmark of 206 tasks across 11 high-value domains where the top model, Claude Opus 5.5, scores just 12%

Key Takeaways

  • Scale: 11 domains, 14 data partners and 206 tasks, with a default time limit of 8 hours per task (GDPx site)
  • Leaderboard (as reported by SemiAnalysis): Claude Opus 5.5 12.0%, Gemini 4 Carbon 10.9%, GPT-6 Astra 9.6%, Muse Wonder Full 9.0%, Claude Fable 5.1 8.5%, GPT 6.1 Sol 8.4%, Claude Sonnet 5.5 8.2%
  • Lower tier and open weights: DeepSeek 4.1 Flash 1017 5.4%, GPT-6 Luna 5.1%, Grok 4.7 3.9%, GLM 5.3 2.7%; almost every model scores 0 on finance and legal
  • Method: each model runs in its native harness where one exists (Claude Code for Opus, Codex for Astra, Antigravity for Carbon), otherwise OpenCode; the headline is pass@1 over three attempts with all domains weighted equally (SemiAnalysis blog)
  • By domain: Opus 5.5 wins quant trading, hardware and chip design; Gemini 4 Carbon wins biology and software engineering; GPT-6 Astra wins RSI and materials science; Muse Wonder Full is the strongest cyber model
Read full story
Linked Models & Agents:
⚡ Claude Opus 5.5⚡ Gemini 4 Carbon⚡ GPT-6 Astra⚡ Muse Wonder Full⚡ DeepSeek 4.1 Flash 1017⚡ GLM-5.3🤖 Claude Code🤖 Codex🤖 Antigravity🤖 OpenCode🌐 SemiAnalysis🌐 Anthropic🌐 Google🌐 OpenAI🌐 Meta🌐 DeepSeek
#GDPx#SemiAnalysis#Benchmark#Agents#GDPval
A
Artificial AnalysisArtificialAnlys·
👀32
🔥81
View on 𝕏 ↗

Artificial Analysis releases AA-Omniscience v1.1: 3,000 refreshed questions and a calibrated 300-question public set, with Gemini 4 Carbon leading at a Net Score of 54

On October 31, independent evaluator Artificial Analysis released AA-Omniscience v1.1, its knowledge and hallucination benchmark and part of Intelligence Index v5. The question set shrinks from 6,000 to 3,000, dropping previously public and saturated questions, fixing errors and ambiguities, and adding new ones, and a new 300-question public set on Hugging Face is calibrated to the private set's difficulty, with 0.99 Pearson and Spearman correlation on Net Score. Gemini 4 Carbon (high) leads with a Net Score of 54, ahead of Claude Opus 5.5 at 48 and Claude Fable 5.1 at 46; Fable 5.1 has the highest accuracy (68%) and Gemini 4 Argon the lowest hallucination rate (4%). Models score about 1.9 points lower than on v1 on average, and MiniMax M3.1 Flash drops 22 points. All figures are as reported by Artificial Analysis.

Artificial Analysis releases AA-Omniscience v1.1: 3,000 refreshed questions and a calibrated 300-question public set, with Gemini 4 Carbon leading at a Net Score of 54

Key Takeaways

  • The question set shrinks from 6,000 to 3,000 across 6 domains and 42 topics: 600 each for science/engineering/math and software engineering, 450 each for finance, health, humanities/social sciences and law
  • A new 300-question public set is on Hugging Face, with 0.99 correlation to private-set Net Score and a median difference of 0.0 points
  • Net Score ranking: Gemini 4 Carbon (high) 54, Claude Opus 5.5 48, Claude Fable 5.1 46; the best open-weights model, Muse Spark Open, trails Gemini 4 Carbon by 28 points
  • Claude Fable 5.1 has the highest accuracy (68%), followed by Claude Opus 5.5 (66%); Gemini 4 Argon has the lowest hallucination rate (4%)
  • Rank order keeps a 0.98 Spearman correlation with v1, but scores are 1.9 points lower on average; MiniMax M3.1 Flash falls from 16 to -6 and MiMo V2.6 Pro from 8 to -5 (vendor-reported)
Read full story
Linked Models & Agents:
⚡ Gemini 4 Carbon⚡ Claude Opus 5.5⚡ Claude Fable 5.1⚡ Gemini 4 Argon⚡ Muse Spark Open⚡ MiniMax-M3.1-Flash⚡ MiMo V2.6 Pro🌐 Artificial Analysis🌐 Google🌐 Anthropic
#AA-Omniscience#Hallucination#Benchmark#Artificial Analysis#Intelligence Index v5
A
Aravind SrinivasAravSrinivas·
👀28
🔥80
View on 𝕏 ↗

Perplexity CEO shares early Gemini 4 Carbon evals: a record 0.857 on the WANDR deep-research benchmark at $5.86 per task, ahead of Argon and GPT-6 Astra

Perplexity CEO Aravind Srinivas posted early evals of Gemini 4 Carbon on X: it scored a record 0.857 on Perplexity's Wide-And-Deep-Research (WANDR) long-form research benchmark at $5.86 per task, versus 0.802 ($7.88) for Gemini 4 Argon and 0.735 ($13.34) for GPT-6 Astra. He called Carbon the strongest model Perplexity has seen for long-form research and answer quality, and said it will come to all Pro and Max users inside Perplexity Computer once there is enough token capacity.

Perplexity CEO shares early Gemini 4 Carbon evals: a record 0.857 on the WANDR deep-research benchmark at $5.86 per task, ahead of Argon and GPT-6 Astra

Key Takeaways

  • WANDR scores (per Perplexity): Gemini 4 Carbon 0.857, Gemini 4 Argon 0.802, GPT-6 Astra 0.735, GPT-6.1 Sol 0.654, Claude Opus 5.5 0.647, Claude Fable 5.1 0.601
  • Cheaper too: Carbon costs $5.86 per task, below Argon at $7.88 and GPT-6 Astra at $13.34
  • Cost frontier in Perplexity's chart: DeepSeek V4 Pro 0813 (0.359, $0.75), GPT-6 Sol (0.566, $2.23), GPT-6.1 Sol (0.654, $3.70) and Carbon
  • Claude Sonnet 5.5 scored 0.550 ($2.73) and Grok 4.7 scored 0.528 ($9.30)
  • Not live yet: Carbon reaches Perplexity Pro and Max users only once token capacity allows, and Google still limits early access to cyber defenders in its Fairwind program
Read full story
Linked Models & Agents:
⚡ Gemini 4 Carbon⚡ Gemini 4 Argon⚡ GPT-6 Astra⚡ GPT-6.1 Sol⚡ GPT-6 Sol⚡ Claude Opus 5.5⚡ Claude Sonnet 5.5⚡ Claude Fable 5.1⚡ Grok 4.7⚡ DeepSeek V4 Pro 0813🤖 Perplexity Computer🌐 Perplexity🌐 Google🌐 OpenAI🌐 Anthropic🌐 xAI🌐 DeepSeek
#Benchmark#deep research#WANDR#Gemini 4 Carbon#Perplexity#cost per task
ADSponsored
C
Cognition@cognition·
👀44
🔥80
View on 𝕏 ↗

Cognition scores Gemini 4 Carbon at 40% on FrontierCode 2.0 for $4.04 a task, near the top, while Gemini 4 Argon gets 26.3% at $6.41

Cognition, the company behind Devin, posted on X at 01:45 UTC on Oct 30 that Google's new Gemini 4 Carbon scores 40% on its FrontierCode 2.0 coding benchmark at $4.04 per task, near the top and behind only Claude Opus 5.5, GPT-6 Astra and Claude Sonnet 5.5. The earlier Gemini 4 Argon scores 26.3% at a higher $6.41 per task. The results are Cognition's own runs, and the leaderboard on its site has been updated.

Cognition scores Gemini 4 Carbon at 40% on FrontierCode 2.0 for $4.04 a task, near the top, while Gemini 4 Argon gets 26.3% at $6.41

Key Takeaways

  • Gemini 4 Carbon scores 40% on FrontierCode 2.0 at $4.04 per task, behind Claude Opus 5.5 (45.5%), GPT-6 Astra (42.8%) and Claude Sonnet 5.5 (post)
  • Gemini 4 Argon scores 26.3% at $6.41 per task, ahead of Muse Wonder 1.0 but behind Claude Haiku 5.5 and Grok 4.7
  • From Argon to Carbon the score rises about 13.7 points while cost per task drops about $2.37
  • The changelog shows Gemini 4 Carbon was added on Oct 28; FrontierCode 2.0 has 132 tasks scored on whether a maintainer would merge the PR (leaderboard)
  • All figures are Cognition's own runs; Google has not published FrontierCode results itself
Read full story
Linked Models & Agents:
⚡ Gemini 4 Carbon⚡ Gemini 4 Argon⚡ Claude Opus 5.5⚡ GPT-6 Astra⚡ Claude Sonnet 5.5⚡ Claude Haiku 5.5⚡ Grok 4.7⚡ Muse Wonder 1.0🤖 Devin🌐 Cognition🌐 Google🌐 Anthropic🌐 OpenAI🌐 xAI🌐 Meta
#FrontierCode#coding benchmark#Cognition#Gemini 4 Carbon#Gemini#AI coding#Mergeability
G
Google DeepMind@GoogleDeepMind·
👀82
🔥92
View on 𝕏 ↗

Google releases Gemini 4 Carbon two weeks after Argon: 59 and first place on the Artificial Analysis Intelligence Index v5, state-of-the-art cybersecurity and long-horizon coding, with early access for cyber defenders in the Fairwind program

Google DeepMind announced Gemini 4 Carbon on Oct 28, calling it its strongest model, just two weeks after Gemini 4 Argon. The official blog says it ranks first on v5.0 of the Artificial Analysis Intelligence Index at 59 and leads three cybersecurity evaluations: 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation and 71.0% on CWE-bench for patching; it also gains 89 points to rank fourth on WebDev Arena at 1769. Access starts this week for a subset of cyber defenders in the Fairwind program, with developers, businesses and consumers to follow. Introductory pricing matches Argon at $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 afterwards.

Google releases Gemini 4 Carbon two weeks after Argon: 59 and first place on the Artificial Analysis Intelligence Index v5, state-of-the-art cybersecurity and long-horizon coding, with early access for cyber defenders in the Fairwind program

Key Takeaways

  • Positioning: Google calls it its strongest model, aimed at cybersecurity and long-horizon agentic work; internal teams saw it run for days to root-cause and patch subtle bugs across large codebases and low-level systems
  • Official scores (evaluated without guardrails): 88.2% on SEC-bench Pro, 54.8% on ExploitGym and 71.0% on Collinear's CWE-bench; 59 and first place on the Artificial Analysis Intelligence Index v5.0; +89 to fourth place on WebDev Arena at 1769
  • Safety and alignment: successful cheating on Terminal-Bench 4.0 fell to 0.0% with a 0.4% attempt rate, high-severity misalignment alerts in internal live coding dropped by almost 80%, unsupported claims are about half of Argon's, and the Gray Swan indirect prompt injection attack success rate fell from Argon's 0.7% to 0.0%
  • Internal examples: more than 2x Argon's success rate at generating secure code, and the Chrome team reports more than 10x better high-quality patches than another lab's large frontier model
  • Rollout: a subset of cyber defenders in the Fairwind program first, alongside the U.S. government's voluntary pre-release testing; no developer API yet; introductory pricing of $2 / $10 per million tokens with cached input 95% off, then $4 / $20
Read full story
Linked Models & Agents:
⚡ Gemini 4 Carbon⚡ Gemini 4 Argon🌐 Google🌐 Google DeepMind🌐 Artificial Analysis
#Google#Gemini#Gemini 4 Carbon#Cybersecurity#long-horizon agents#frontier model
You've caught up with all stories