AICoder

AI Coding Live News

First-hand AI coding updates from 𝕏/Twitter frontier labs and researchers, distilled by AI with entity benchmarks.

Explore live on 𝕏 ↗

AI Coding Live News

S
SemiAnalysis@SemiAnalysis_·
👀94
🔥85
View on 𝕏 ↗

SemiAnalysis releases GDPx, an agentic benchmark of 206 tasks across 11 high-value domains where the top model, Claude Opus 5.5, scores just 12%

On Nov 2, semiconductor and AI research firm SemiAnalysis released GDPx, a benchmark of agentic capability across 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement (RSI), hardware design, materials science, alignment and software engineering. Its 206 tasks were built by 14 specialist data partners, with SemiAnalysis doing final QA. On the official leaderboard Claude Opus 5.5 leads at 12.0%, followed by Gemini 4 Carbon at 10.9% and GPT-6 Astra at 9.6%; nearly every model scores 0 on finance and legal. SemiAnalysis says safety classifiers drag down Opus 5.5 and GPT-6 Astra, and plans a new version every 3–6 months. All scores are as reported by SemiAnalysis.

SemiAnalysis releases GDPx, an agentic benchmark of 206 tasks across 11 high-value domains where the top model, Claude Opus 5.5, scores just 12%

Key Takeaways

  • Scale: 11 domains, 14 data partners and 206 tasks, with a default time limit of 8 hours per task (GDPx site)
  • Leaderboard (as reported by SemiAnalysis): Claude Opus 5.5 12.0%, Gemini 4 Carbon 10.9%, GPT-6 Astra 9.6%, Muse Wonder Full 9.0%, Claude Fable 5.1 8.5%, GPT 6.1 Sol 8.4%, Claude Sonnet 5.5 8.2%
  • Lower tier and open weights: DeepSeek 4.1 Flash 1017 5.4%, GPT-6 Luna 5.1%, Grok 4.7 3.9%, GLM 5.3 2.7%; almost every model scores 0 on finance and legal
  • Method: each model runs in its native harness where one exists (Claude Code for Opus, Codex for Astra, Antigravity for Carbon), otherwise OpenCode; the headline is pass@1 over three attempts with all domains weighted equally (SemiAnalysis blog)
  • By domain: Opus 5.5 wins quant trading, hardware and chip design; Gemini 4 Carbon wins biology and software engineering; GPT-6 Astra wins RSI and materials science; Muse Wonder Full is the strongest cyber model
Read full story
Linked Models & Agents:
⚡ Claude Opus 5.5⚡ Gemini 4 Carbon⚡ GPT-6 Astra⚡ Muse Wonder Full⚡ DeepSeek 4.1 Flash 1017⚡ GLM-5.3🤖 Claude Code🤖 Codex🤖 Antigravity🤖 OpenCode🌐 SemiAnalysis🌐 Anthropic🌐 Google🌐 OpenAI🌐 Meta🌐 DeepSeek
#GDPx#SemiAnalysis#Benchmark#Agents#GDPval
A
Anthropic@Anthropic·
👀101
🔥91

Anthropic details Claude anti-distillation stack: two-stage classifiers and preserved thinking cut harvested reasoning tokens ~10–100×

On 2026-10-12 Anthropic published a research post claiming substantially stronger defenses against illicit distillation of Claude: early evidence of ~90–99% less data harvested by distillers, and ~10–100× fewer reasoning tokens extractable on-platform after new safeguards. Methods include preserved thinking (harder context rewriting for newer API accounts), org-scoped reasoning transcripts, a two-stage classifier against chain-of-thought extraction (<0.1% claimed false-positive rate), bans/limits, and resumed charging for blocked low-FPR requests. The post cites cyber-capability distillation into models such as GLM-5.3 and urges other frontier labs and platforms to share the load.

Key Takeaways

  • Official: Anthropic Research — distillation defenses (2026-10-12)
  • Vendor effect: ~90–99% less data harvested; ~10–100× fewer reasoning tokens extractable on-platform; tens of thousands of accounts banned, often within ~2 hours
  • Stack: preserved thinking (newer API accounts), org-scoped reasoning transcripts, two-stage CoT-extraction classifiers (<0.1% claimed FPR)
  • Coverage: Sonnet 5.5, Opus 5/5.5, Fable 5/5.1 among generally available targets; legitimate apps may need integration updates and can hit rare charged blocks
  • Industry: cites open-weight cyber-capability distillation cases; argues platforms must disrupt fraudulent proxies too
Read full story
Linked Models & Agents:
⚡ Claude Sonnet 5.5⚡ Claude Opus 5.5⚡ Claude Fable 5.1🌐 Anthropic🌐 Zhipu
#Anthropic#Claude#Distillation#Safety#Preserved Thinking#API#Security
C
Confident AI@confident-ai·
👀73
🔥90

DeepEval 4.2: Jev for LLM-as-judge decisions at ~$0.042/M input tokens and ~100ms

Confident AI’s deepeval 4.2.x (notes for 4.2.2/4.2.4; PyPI at 4.2.6) wires TypeSafe Jev into LLM-as-a-judge: evaluation LLMs still handle language tasks, while closed verdicts go to System One for typed decisions with calibrated probabilities. Claimed decision-stage pricing is about $0.042 per million input tokens with no output charge, ~100ms typical latency, and batched independent questions. Faithfulness, GEval, DAGMetric, and Classifiers are integrated; enable via deepeval set-mode experimental plus typesafe-sdk—missing keys fail loudly instead of silent fallback.

DeepEval 4.2: Jev for LLM-as-judge decisions at ~$0.042/M input tokens and ~100ms

Key Takeaways

  • Notes: python-v4.2.4; blog Introducing Jev for Evals; PyPI deepeval 4.2.6
  • Official: ~$0.042/M input tokens for decisions, no output charge; ~100ms typical; batch independent questions
  • Faithfulness/GEval/DAGMetric keep language work on the judge LLM; closed verdicts use Jev Noul/Choice/Score
  • New Classifiers for categorical labels (e.g. RefusalClassifier)
  • Opt-in: deepeval set-mode experimental + typesafe-sdk + TYPESAFE_API_KEY (loud failure if missing)
Read full story
Linked Models & Agents:
⚡ Jev⚡ TypeSafe-System-One🤖 DeepEval🤖 LLM-as-Judge🌐 Confident-AI🌐 TypeSafe🌐 DeepEval
#DeepEval#Jev#TypeSafe#LLM-as-judge#Faithfulness#GEval#Classifiers#Evals#CI
H
Hugging Face@huggingface·
👀77
🔥94

Just Ask Jev: Zero-Shot AI Alignment Failure Detector via Calibrated RL Slashes LLM Judge Evaluation Cost by 63x

Safety researchers have introduced 'Just Ask Jev' alongside the RLCDAlignBench suite (arXiv: 2609.29429, GitHub: sumleo/RLCDAlignBench). Conventional alignment auditors rely on expensive generative LLM judges or rigid single-label classifiers (e.g., Llama Guard). Trained via Reinforcement Learning for Calibrated Decisions (RLCD), Jev evaluates multiple typed alignment questions simultaneously across ten failure modes—sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy breaches, bias, reward hacking, uncertainty concealment, and power seeking—in a single forward pass. Achieving a median 0.886 zero-shot AUROC across 44 benchmarks, Jev matches human consensus while slashing evaluation costs by 63x.

Just Ask Jev: Zero-Shot AI Alignment Failure Detector via Calibrated RL Slashes LLM Judge Evaluation Cost by 63x

Key Takeaways

  • Evaluates ten distinct alignment failure modes simultaneously with calibrated probability outputs in a single forward pass.
  • Achieves a 0.886 median AUROC in zero-shot settings across 44 benchmark suites and five frontier target LLMs on RLCDAlignBench.
  • Reduces deployment and audit inference expenditure by 63x compared to text-generating LLM-as-a-judge pipelines.
  • Outperforms supervised domain-specific classification baselines while surfacing subtle labeling defects in legacy datasets.
  • Full evaluation harness, calibrated inference code, and annotated benchmarks open-sourced on GitHub.
Read full story
Linked Models & Agents:
⚡ Jev Model⚡ Llama Guard🤖 safety-agent🤖 alignment-evaluator🌐 huggingface
#AI Alignment#Safety#RLCD#Guardrails#Benchmarking#Open Source#Hugging Face
ADSponsored
P
Pipecat@pipecat_ai·
👀56
🔥92

Pipecat 1.12.0: cross-vendor IPA pronunciation transforms, MOQTransport client reconnect, Classifier/LLMClassifier eval surface

Pipecat v1.12.0 (2026-09-26) adds TTSService.pronunciation_transform_ipa() for reusable word→IPA maps across Cartesia/ElevenLabs/Inworld/Deepgram, client-mode MOQTransport reconnect with session-ending markers, Pipecat Classifiers + LLMClassifier, frame-level interruptible, and response_schema on LLM run_inference. PyPI pipecat-ai 1.12.0 is live.

Pipecat 1.12.0: cross-vendor IPA pronunciation transforms, MOQTransport client reconnect, Classifier/LLMClassifier eval surface

Key Takeaways

  • Shipped: GitHub v1.12.0 = PyPI pipecat-ai 1.12.0; docs docs.pipecat.ai
  • Pronunciation: pronunciation_transform_ipa maps words to IPA once; vendors emit phoneme/SSML/slash IPA—register last in text_transforms
  • Transport: MOQTransport client reconnect within connection_timeout (default 60s); session-ending distinguishes hangup vs outage; relay_url full URL
  • Eval: Classifiers (YesNo/Choice/Score) + LLMClassifier batch questions; allow_continue=false for final-answer judges
  • Inference: run_inference response_schema enforced by OpenAI/Anthropic/Google; frames interruptible by default; UninterruptibleFrame deprecated
Read full story
Linked Models & Agents:
⚡ grok-voice-transcribe-2.0🤖 pipecat🌐 Daily🌐 Cartesia🌐 ElevenLabs🌐 Deepgram🌐 xAI
#Pipecat#voice agent#TTS#IPA#MOQ#Classifier#Realtime#Python
M
Mastra@mastra_ai·
👀54
🔥90

Mastra @mastra/core 1.70.0: ModelSelectionProcessor routes cheap vs strong models; durable streams gain closeOnSuspend

Mastra published @mastra/[email protected] on 2026-09-24 with ModelSelectionProcessor for classifier-backed per-request model routing (#24865), closeOnSuspend on durable agent streams so AG-UI-style loops can finish on tool suspend, plus createClassifierScorer(), optional tool titles, and trace aggregate dimension/measure helpers with tag predicates. Docs warn routing is not always cheaper because models do not share prompt caches.

Key Takeaways

  • ModelSelectionProcessor routes simple vs hard requests (#24865); scope: 'first-step' optional
  • Durable closeOnSuspend: true ends streams at tool suspend (#24894)
  • createClassifierScorer() maps Classifier questions to 0–1 scores (#24792)
  • Trace aggregate descriptors + tag predicates for advanced queries
  • Upgrade: npm i @mastra/[email protected]; see CHANGELOG + processors docs
Read full story
Linked Models & Agents:
🤖 Mastra🌐 Mastra
#Mastra#TypeScript#ModelSelectionProcessor#durable agents#Classifiers#Observability
M
Mastra@mastra_ai·
👀52
🔥90

Mastra @mastra/core 1.69.0: first-class classifiers, fail-closed ClassifierProcessor, background.adopt, 10 Connect providers

Mastra shipped @mastra/[email protected] on 2026-09-24: register classifiers on new Mastra({ classifiers }) and use them as typed workflow steps; ClassifierProcessor fails closed by default on I/O/stream; tools can context.background.adopt() long work; @mastra/connect adds ten generated SaaS tool providers (Slack, GitHub, Gmail, …).

Mastra @mastra/core 1.69.0: first-class classifiers, fail-closed ClassifierProcessor, background.adopt, 10 Connect providers

Key Takeaways

  • @mastra/[email protected] tagged 2026-09-24 06:58 UTC (npm also rolled 1.70.0 later that day)
  • First-class classifiers with list/add/remove and typed workflow steps + token usage
  • ClassifierProcessor fails closed by default; opt into errorStrategy: 'warn'
  • context.background.adopt({ completion, cancel }) for ack-then-finish tools (in-memory handles)
  • @mastra/[email protected] ships 10 generated providers (Slack, GitHub, Gmail/Calendar, …)
Read full story
Linked Models & Agents:
🤖 Mastra🌐 Mastra
#Mastra#TypeScript#Classifiers#ClassifierProcessor#Connect#Inngest#Agents
O
OpenAI@OpenAI·
👀47
🔥96
View on 𝕏 ↗

OpenAI: Astra hits Critical cyber threshold; release with stronger safeguards

OpenAI said upcoming Astra is the first model it designates at the Critical cybersecurity capability threshold under its Preparedness Framework, beating GPT-5.6 Sol on exploit benchmarks and even discovering zero-days in evals. It delayed parts of training/release, hardened refusals, classifiers, and chain-of-thought monitoring; advanced cyber access will start with limited testers and Daybreak Blue.

Key Takeaways

  • Astra is OpenAI's first Critical cybersecurity designation under the Preparedness Framework.
  • Perfect ExploitBench score; far stronger than GPT-5.6 Sol on an internal high-severity V8 set, including two zero-days.
  • Hardened refusals (~91.5% jailbreak refusal), classifiers, and CoT monitoring; advanced cyber access starts limited.
Read full story
Linked Models & Agents:
⚡ GPT-5.6 Sol🌐 OpenAI
#Astra#Preparedness Framework#Critical cyber#ExploitBench#Daybreak Blue#Agentic coding#OpenAI
You've caught up with all stories