AI Coding Live News

Perplexity Cuts Agent API Effort-Level Costs and Adds Ultra: Medium Nearly 90% Cheaper; Ultra ~85.9% WANDR

On November 5 Perplexity Developers (@perplexitydevs) said every Agent API effort level now costs less per query with equal or better research quality, and added an ultra tier for broad long-running research. Vendor figures: medium rose from ~55% to ~77% on BrowseComp/DSQA with nearly 90% lower cost; xhigh held ~76.5% WANDR at ~25% of August 2026 cost; ultra runs three parallel researchers plus a consolidator at ~85.9% WANDR and about $21.71/task.

Perplexity Cuts Agent API Effort-Level Costs and Adds Ultra: Medium Nearly 90% Cheaper; Ultra ~85.9% WANDR

Key Takeaways

  • Official (@perplexitydevs + blog): every effort level cheaper with equal/better quality; new ultra
  • Vendor: medium BrowseComp/DSQA ~55%→~77% with nearly 90% lower cost
  • Vendor: xhigh WANDR ~76.5% at ~25% of August 2026 cost
  • ultra: 3 parallel research agents + consolidator; WANDR ~85.9% at ~$21.71/task on Claude Opus 5.5
  • Use effort="ultra" or pin [email protected]; background mode for long runs
Read full story

Thinking Machines Launches Inkling 2 Small: 594B/17B MoE, Vendor-Reported AA Intelligence Index ~31; Tinker Inference and Fine-Tuning Now, Apache 2.0 Weights Planned Next Week

On November 5 Thinking Machines Lab introduced Inkling 2 Small, the first second-generation model: a 594B-total / 17B-active MoE with image input and variable thinking effort. The company reports an Artificial Analysis Intelligence Index score of about 31, with large gains versus prior Inkling models in agentic coding, vision and knowledge work. It is available now on Tinker for inference and fine-tuning with closed weights; full Apache 2.0 weights are planned next week after external safety testing. Tinker list pricing is about $0.30/$1.00 per million input/output tokens at 64K context.

Thinking Machines Launches Inkling 2 Small: 594B/17B MoE, Vendor-Reported AA Intelligence Index ~31; Tinker Inference and Fine-Tuning Now, Apache 2.0 Weights Planned Next Week

Key Takeaways

  • Specs (official): 594B total / 17B active MoE; Hybrid + Vision; variable effort; no audio (unlike prior Inkling)
  • Scores (vendor): AA Intelligence Index ~31; FrontierCode v2.0 ~25.4%; DeepSWE v1.1 ~72.1%; DesignArena Website Elo ~1340
  • Availability: Tinker Playground / fine-tune and sampling today; closed weights; Apache 2.0 full weights planned next week after safety tests
  • Pricing (Tinker docs): Prefill $0.30 / Sample $1.00 / Train $1.50 per 1M tokens at 64K; cached Prefill $0.03
  • Training note (official): joint RL ~1,075 steps and 17M+ rollouts; early checkpoints tested on OpenCode as Fledge-Alpha
Read full story
Linked Models & Agents:
⚡ Inkling 2 Small⚡ Inkling-Small⚡ Inkling🤖 OpenCode🤖 Tinker🌐 Thinking Machines

Artificial Analysis Makes Time per Task Its Headline Performance Metric: GPT-6 Astra Finishes a Max-Effort Task in About 17 Minutes, Ahead of Opus 5.5 and Gemini 4 Carbon

Independent benchmarker Artificial Analysis announced on X on November 4 (17:34 UTC) a new version of its API performance benchmarking: the headline metric moves from output speed to Time per Task, the time to finish one Intelligence Index task (output tokens per task divided by end-to-end output speed). It covers about 1,000 serverless endpoints, measures speed from sending the request to the final token, and adds multi-turn agentic coding and knowledge-work workloads. Per its data, GPT-6 Astra at max effort runs at 51 tokens/s yet finishes a task in about 17 minutes, against about 26 for Claude Opus 5.5 (max) and about 29 for Gemini 4 Carbon (high).

Key Takeaways

  • Time per Task equals output tokens per Intelligence Index task divided by end-to-end output speed, so a model that is fast per token but verbose can finish later than a slower model that uses fewer tokens.
  • End-to-end output speed runs from sending the request to the final token, including the wait before the first token (prefill and hidden reasoning); output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer.
  • GPT-6 Astra (max) runs at 51 tokens/s with about 50k output tokens per task and takes about 17 minutes; Claude Opus 5.5 (max, 111 tokens/s) takes about 26 minutes and Gemini 4 Carbon (high, 71 tokens/s) about 29, using roughly 120k to 170k tokens per task.
  • At each lab's API default effort (medium for all three), GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 cut Time per Task by about 3 to 5x and Cost per Task by about 2 to 4x, scoring about 5 to 11 points lower; Opus 5.5 drops from about 26 minutes and $10.04 to about 6.8 minutes and $2.45.
  • The official methodology page lists this as version 3.0.0 (October 30), with runs now every 6 hours and a new endpoint error rate metric.
Read full story

X discussion: Shopify CEO Tobi Lütke backs the monorepo as a Shopify engineer calls it a boon for coding agents, while critics still prefer split repos

Shopify distinguished engineer Burke Libbey posted on November 1 that moving to a monorepo has been a boon for agents at Shopify; it needs thoughtful tooling and has downsides that take effort to mitigate, but he would not go back. He was replying to Darren Shepherd, who said a day earlier that he hates the idea that monorepos are great for agents. Shopify CEO Tobi Lütke then quoted Burke and recalled Mikhail Parakhin telling him that every engineer hates the idea of moving to a monorepo, yet once it is there no engineer would want to work somewhere without one. Tobi's post drew about 1,115 likes, 454 bookmarks and 89K views and set off a wide debate. These are practitioner opinions, not an official technical announcement.

Key Takeaways

  • Original post: @tobi, Nov 1, 15:39 UTC, about 1,115 likes, 454 bookmarks, 95 replies and 89K views at fetch time
  • Shopify distinguished engineer Burke Libbey wrote that moving to a monorepo has been a boon for agents at Shopify; it needs thoughtful tooling and its downsides take effort to mitigate, but he would not go back
  • On the other side, Darren Shepherd (Rancher and k3s background) posted on Oct 31 that he hates the idea that monorepos are great for agents and that the best way to scale is to avoid scaling
  • Mikhail Parakhin replied to Tobi's retelling: "True story! And now, with agents, it more true than ever."
  • Supporters in the quote posts say agents can see every caller at once and get cross-stack type checking and atomic changes; skeptics ask whether agents make independent, separately evolving repos more attractive
Read full story
Linked Models & Agents:
ADSponsored

X discussion: AP reports AI companies are turning to religion to teach chatbots right from wrong, with Anthropic citing talks with 15+ religious and cross-cultural groups

A topic drawing heavy discussion on X: a new Associated Press feature reports that some AI companies are turning to religion as they work out how to teach chatbots right from wrong. On Nov 1 the market-news account unusual_whales relayed the report, noting that Anthropic says it has met with scholars, clergy, philosophers and ethicists from more than 15 religious and cross-cultural groups to help shape how Claude handles moral questions. The post drew about 880 likes, 162K views and 189 replies, and X listed the story among its trending news topics. The Anthropic effort itself is not new: the company described it in an official post on May 19. This is a write-up of the discussion, with opinions presented as opinions.

Key Takeaways

  • Buzz: the unusual_whales post has about 879 likes, 189 replies, 66 quotes and 162K views (public X metrics as of Nov 1, 13:10 SGT), and X lists the story as a trending news topic
  • Source: an Associated Press feature whose lede quotes Anthropic co-founder Chris Olah at the Vatican presentation of Pope Leo XIV's AI encyclical: "These are not questions a lab can answer"
  • Official background: Anthropic's May 19 post Widening the conversation on frontier AI says its first round of dialogues was with scholars, clergy, philosophers and ethicists from more than 15 religious and cross-cultural groups, and stresses it is not aligning models with any one tradition's worldview
  • Concrete experiment: inspired by the idea of a mentor as an external conscience, Anthropic gave Claude a tool it can call mid-task that returns a reminder of its ethical commitments, and says this markedly lowered misaligned behavior on several internal alignment evals (vendor-reported, no figures published)
  • Debate: some welcome drawing on many traditions, others worry about importing religious disagreements into AI, and one reply joked that refusals will now sound genuinely disappointed in you
Read full story
Linked Models & Agents:
⚡ Claude🌐 Anthropic
A
Anthropic@AICoder·
👀104
🔥80

Claude Code 2.1.314 adds a block on writes outside working directories, IdP group routing for the apps gateway, and idle compaction in VS Code and Desktop

Anthropic published Claude Code v2.1.314 on GitHub at 01:06 UTC on Oct 30. The official CHANGELOG lists 119 entries, more than half of them fixes. The new features lean enterprise: permissions.blockWritesOutsideWorkingDirectories makes Write, Edit and NotebookEdit refuse paths outside the working directories in every permission mode; Claude apps gateway upstreams can be limited to IdP groups, and Bedrock upstreams can log which guardrail fired; idle compaction now also runs in VS Code and the Code tab of Claude Desktop. It also fixes sessions failing repeatedly when an admin caps effort, and claude -p exiting 0 with no output when startup failed.

Key Takeaways

  • Size: the official v2.1.314 CHANGELOG has 119 entries, more than half of them fixes; released at 01:06 UTC on Oct 30
  • Safety: the new permissions.blockWritesOutsideWorkingDirectories setting stops Write, Edit and NotebookEdit from writing outside the working directories in every permission mode
  • Enterprise gateway: upstreams take an optional groups list so only developers in those IdP groups are routed there, failover included; Bedrock upstreams gain guardrail_trace
  • Experience: idle compaction now runs in VS Code and the Claude Desktop Code tab, summarizing idle conversations above 200K tokens
  • Fixes: when an admin caps effort, Claude Code lowers it and retries; a failed claude -p startup now prints the error and exits 1
Read full story
Linked Models & Agents:

Claude Code lead Boris Cherny on faster usage drain: every shared session reviewed, no regression found, fixes for a few smaller issues rolling out

Since Oct 21 many developers on X have complained that Claude Code was draining usage limits much faster. On Oct 28 (UTC) Boris Cherny, who leads Claude Code at Anthropic, replied on X that the team reviewed every session ID users shared, found no single root cause and no regression in Claude Code, and is rolling out fixes for a few smaller issues that made some sessions burn tokens faster. Most fast-drain cases traced back to one or two heavy sessions using xhigh effort, /goal, Workflows, Agent Teams or long SDK jobs.

Key Takeaways

  • The complaint post that started it said usage was draining 5x faster and drew about 1,660 likes and 240+ replies (a user claim, not official data)
  • Boris Cherny: the team checked every shared session ID, found no single root cause or Claude Code regression, and is rolling out fixes for a few smaller issues
  • Heavy features were the main driver: xhigh effort, /goal, Workflows, Agent Teams and long SDK jobs
  • Subagents inherit the session effort level by default, which adds up fast; /usage shows where tokens went
  • The official docs say Agent Teams use about 7x more tokens than standard sessions when teammates run in plan mode
Read full story
Linked Models & Agents:

GitHub Universe 2026: Copilot Fusion is generally available, routing and orchestrating models automatically, with Fusion Max scoring 60 on the Artificial Analysis Coding Agent Index at about half the cost of Claude Code with Opus 5.5 Max

At GitHub Universe 2026 on Oct 28, GitHub made Copilot Fusion generally available. It merges the multi-model orchestration of the Project HydraFusion research preview with Copilot auto model selection: users pick Copilot Fusion and an effort level (Standard, High or Max) in the model picker, and the system decides which models to use and whether to run a Single, Cascade or Critique workflow. GitHub says Fusion Max scored 60 on the Artificial Analysis Coding Agent Index, two points below Claude Code with Opus 5.5 Max, at about half the API cost per task, about half the tokens and about 40% less time. Each underlying model is billed at its standard rate, and paid plans get 30% off through Nov 30.

GitHub Universe 2026: Copilot Fusion is generally available, routing and orchestrating models automatically, with Fusion Max scoring 60 on the Artificial Analysis Coding Agent Index at about half the cost of Claude Code with Opus 5.5 Max

Key Takeaways

  • Scores (Artificial Analysis Coding Agent Index v2.0): Fusion Max 60 versus 62 for Claude Code with Opus 5.5 Max; $5.59 per task versus $11.42, while the Devin Fusion pairing scores 62 at $8.00
  • Breakdown: a slight lead on Terminal-Bench 4.0 at 71, third on FrontierCode at 51 (Claude Code 56, Devin Fusion 54), and a weaker 57 on SWE-Atlas-QnA-AA
  • Efficiency: about 7.2M tokens per task, roughly half of Claude Code with Opus 5.5 and 30% fewer than Devin Fusion; GitHub reports about 40% less time per task
  • The pool includes Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra and GPT-6 Sol; admins can disable individual models for Fusion only and set Fusion as the default
  • Availability: all Copilot plans in VS Code, the Copilot app, Copilot CLI and JetBrains IDEs; existing auto, HydraFusion and HydraFusion Max selections migrate to matching effort levels
Read full story

Mistral launches its Responses API in public preview: OpenAI Responses-compatible, returning the answer, reasoning and tool calls as separate items for building agents

Mistral AI announced on X on Oct 28 that its Responses API is in public preview, positioned as the interface for building agents on Mistral. The official docs say it uses the OpenAI Responses format, and each response's output array carries the models messages, its reasoning and its function calls as separate items, which is easier to handle step by step than Chat Completions. The preview is stateless, so the app keeps the conversation history and sends it back; it supports text, image and file input, JSON Schema structured output, streaming events, client-executed function calls, reasoning.effort, and token counting via POST /v1/responses/input_tokens. It is served in the EU and US on Mistral AI Cloud Inference. The same thread says Mistral now also serves DeepSeek V4.1 Flash (public preview) and GLM 5.3 Flash (generally available).

Key Takeaways

  • The Responses API is in public preview and uses the OpenAI Responses format, so existing Responses integrations can move to Mistral more easily
  • Each response splits the answer, reasoning and function calls into separate output items, which suits step-by-step agent loops
  • The preview is stateless: the app keeps the history and sends earlier output items back along with function_call_output results
  • It supports image and file input, JSON Schema structured output, streaming, function calling (including namespaces), reasoning.effort and token counting via /v1/responses/input_tokens
  • It is served in the EU and US on Mistral AI Cloud Inference; Mistral also added DeepSeek V4.1 Flash (preview) and GLM 5.3 Flash (GA)
Read full story

Cursor adds DeepSeek V4.1 Flash: all four effort levels sit on the CursorBench 4.0 cost frontier, with Max scoring 47.7% at $0.98 per task

Cursor announced on Oct 26 (UTC) that DeepSeek V4.1 Flash is now available in Cursor and can be turned on under Cursor Settings > Models. Cursor says every effort level sits on the CursorBench cost frontier, from $0.60 to $0.98 per task, with no model scoring higher at those prices. The official CursorBench 4.0 leaderboard lists DeepSeek V4.1 Flash 1017 at 47.7% on Max ($0.98 per task), 46.3% on Medium ($0.80), 45.6% on High ($0.77) and 44.9% on Low ($0.60). The top score on the board, Opus 5.5 Max, is 57.8% but costs $13.43 per task.

Cursor adds DeepSeek V4.1 Flash: all four effort levels sit on the CursorBench 4.0 cost frontier, with Max scoring 47.7% at $0.98 per task

Key Takeaways

  • Four effort levels on the official CursorBench 4.0 board: Max 47.7% at $0.98, Medium 46.3% at $0.80, High 45.6% at $0.77, Low 44.9% at $0.60; Max ranks 17th overall
  • Value: Max at 47.7% nearly ties Sonnet 5.5 High (47.8%, $1.20) for about 18% less; Opus 5.5 Max scores about 10 points higher but costs $13.43 per task, roughly 13.7x more
  • Nearest rivals on price: Haiku 5.5 Max at 48.4% for $1.12 and Haiku 5.5 Extra High at 44.3% for $0.56, bracketing DeepSeek's price range
  • The trade-off is tokens and steps: Max averages about 537K tokens and 187 steps per task, versus about 37K tokens and 41 steps for Sonnet 5.5 High, so expect longer runs
  • Turn it on under Cursor Settings > Models; a counterintuitive detail is that Medium (46.3%) scores slightly above High (45.6%) for only $0.03 more
Read full story
ADSponsored

OpenAI launches GPT-6.1 Terra: fast, low-cost computer use at $1/$5 per million tokens, half the price of GPT-5.6 Terra and GPT-6.1 Sol

On Oct 26 OpenAI released GPT-6.1 Terra, an efficient model for visual reasoning, precise computer use and natural writing. It is rolling out today to Plus, Pro, Enterprise and Edu users in ChatGPT Work and Codex, and in the API as gpt-6.1-terra. Standard API pricing is $1 input, $0.05 cached input and $5 output per million tokens, which OpenAI says is 50% lower than GPT-5.6 Terra and GPT-6.1 Sol, with a 1,050,000-token context window. By OpenAI's figures it scores 15 points higher than GPT-5.6 Terra on Agents' Last Exam at medium effort for about a third of the cost, and solves 50.3% of Terminal-Bench 4.0 at max effort.

OpenAI launches GPT-6.1 Terra: fast, low-cost computer use at $1/$5 per million tokens, half the price of GPT-5.6 Terra and GPT-6.1 Sol

Key Takeaways

  • Pricing (official docs): $1 input, $0.05 cached input, $1.25 for 30-minute cache writes and $5 output per million tokens; requests over 272K input tokens are billed at 2x input and 1.5x output; Batch/Flex are half price and Fast mode is 2x
  • Specs: 1,050,000-token context, 922,000 max input and 128,000 max output tokens, Jun 1 2026 knowledge cutoff; reasoning.effort from none to max, and tool calling requires the Responses API
  • Computer use (OpenAI figures): on Agents' Last Exam at medium effort it beats GPT-5.6 Terra by 15 points at about one-third the cost, and matches GPT-6 Astra at one-tenth the price per token; it finishes tasks over 3x faster than GPT-6 Astra and GPT-6.1 Sol
  • Coding and factuality (OpenAI figures): 50.3% on Terminal-Bench 4.0 at max effort, more than double GPT-5.6 Terra's published 21.5%; error rate on an internal error-inducing prompt set fell from 53.6% to 16.7% at low effort
  • Safety: OpenAI says it meets the Critical cybersecurity threshold under its Preparedness Framework and keeps the safeguards built for GPT-6 Astra
Read full story

X discussion: Matt Pocock sketches the ideal "language for agents" (Rust speed, Go-sized design space, strictest verification) and names Bend; Chris Lattner replies with Mojo

At 17:23 UTC on 2026-10-25, TypeScript educator Matt Pocock (@mattpocockuk, author of Total TypeScript and AI Hero) posted (post) that the next great "language for agents" should be as fast as Rust, have as little design space as Go, and have the strictest type/verification system ever written, calling Bend "an interesting first effort". When we pulled it, the post had about 3,130 likes, 84 reposts, 264 replies, 786 bookmarks and 220K views, and it reached X's trending stories. Mojo creator Chris Lattner replied (reply) recommending Mojo and suggesting "can target all hardware" be added to the list. This is an opinion debate, not a verified conclusion.

Key Takeaways

  • Main post: post, published 17:23 UTC on 10-25, with about 3,130 likes, 84 reposts, 264 replies, 786 bookmarks and 220K views when we pulled it
  • Three criteria: Rust-level speed, a Go-sized design space (few ways to write the same thing), and the strictest type/verification system ever written, so the compiler rather than a human catches agent mistakes
  • Chris Lattner replied recommending Mojo, citing advanced type features such as linear types and symbolic constraints, and suggested adding "can target all hardware" (reply, about 40 likes when pulled)
  • Bend creator Victor Taelin (@VictorTaelin) also joined the thread, saying Bend blocking an agent from breaking an app is "working as intended"
  • No benchmarks or controlled comparisons were shared; whether Bend or Mojo suits agent coding remains personal opinion
Read full story
Linked Models & Agents:

BOSS Zhipin's Nanbeige lab open-sources Nanbeige4.3-33B, a 33B dense agent model with 512K context under Apache 2.0, reporting 80.9 on Terminal-Bench 2.1 vs 73.0 for Qwen3.8-27B

At 12:11 UTC on 2026-10-25, the Nanbeige LLM Lab of BOSS Zhipin (Kanzhun Limited) announced Nanbeige4.3-33B from its account @nanbeige (post), with the model and its Base version released as open weights on Hugging Face under Apache 2.0 (model card). It is a 33B dense model on a Llama-style architecture with a 512K context window, positioned as the teacher model of the Nanbeige4.3 series that will be distilled into smaller models such as Nanbeige4.3-3B. The official model card compares it with Qwen3.8-27B, Muse-Glimmer-30B and Gemma4-31B using the same harness at highest thinking effort: 80.9 on Terminal-Bench 2.1 (Qwen3.8-27B 73.0) and 65.5 on DeepSWE (42.5), but 60.3 on SWE-Bench Pro, slightly below Qwen's 61.7, and 84.8 on GDPval vs 86.4. All figures are self-reported and not yet independently reproduced.

Key Takeaways

  • Official sources: @nanbeige post on X and the Hugging Face model card; the post had about 616 likes, 79 reposts and 38K views when we pulled it
  • Specs: 33B dense, Llama-style architecture, 512K context, Apache 2.0, with a Base checkpoint released too
  • Self-reported coding scores: Terminal-Bench 2.1 80.9, Terminal-Bench 4.0 23.8, DeepSWE 65.5, SWE-Bench Pro 60.3 (Qwen3.8-27B: 73.0, 6.3, 42.5, 61.7)
  • Self-reported cowork scores: Automation Bench 54.2, JobBench 44.8, GDPval 84.8 (Qwen3.8-27B: 38.3, 34.5, 86.4)
  • Deployment: Transformers, SGLang and vLLM, with official examples using 4-way tensor parallelism; Nanbeige4.3-3B distillation is next
Read full story
Linked Models & Agents:
⚡ Nanbeige4.3-33B⚡ Qwen3.8-27B⚡ Gemma4-31B⚡ Muse-Glimmer-30B🌐 Nanbeige🌐 BOSS Zhipin🌐 Kanzhun

Artificial Analysis Releases HealthBench Medical Evaluation: Verbosity Penalties Reshuffle Ranks as GPT-6 Astra Takes #1

Independent AI evaluation authority Artificial Analysis launched the HealthBench and HealthBench Professional clinical reasoning and medical conversation leaderboards. Addressing the critical flaw where models artificially inflate medical scores through exhaustive, padded responses, the Professional benchmark implements rigorous length penalties. GPT-6 Astra (max effort) leads the world at 66.0%, with GPT-6.1 Sol at 65.1% and Claude Sonnet 5.5 at 63.0%. Claude Opus 5.5, despite holding a 71.4% raw score, collapses to 59.9% (#6) due to averaging ~5,900 characters per reply.

Artificial Analysis Releases HealthBench Medical Evaluation: Verbosity Penalties Reshuffle Ranks as GPT-6 Astra Takes #1

Key Takeaways

  • Artificial Analysis launched HealthBench, introducing strict length penalties on the Professional leaderboard to penalize verbose medical answers
  • GPT-6 Astra took #1 with 66.0%, followed by GPT-6.1 Sol at 65.1% and Claude Sonnet 5.5 at 63.0%
  • Claude Opus 5.5 scored 71.4% raw, but dropped to 59.9% (#6) under length penalties due to averaging ~5,900 characters per reply
  • On standard HealthBench, DeepSeek V4.1 Flash took #2 with 60.9%, outperforming multiple frontier commercial models
Read full story
Linked Models & Agents:

OpenAI open-sources MultiLangBench and reports low-resource language gains: GPT-6 Astra averages 87.4% building SQLite across 20 languages, GPT-6.1 Sol reaches up to 97.9% on COBOL migration

On October 19 OpenAI said its latest models are much better at low-resource languages such as COBOL, Fortran, Visual Basic and OCaml, and open-sourced MultiLangBench, which asks a model to build a SQLite-compatible database engine from scratch in each of 20 languages. GPT-6 Astra averages an 87.4% test pass rate (GPT-5.6 Sol: 26.8%), above 83% in 19 of 20 languages, with COBOL hardest at 72.1%. On a separate COBOL migration eval requiring byte-for-byte output parity after porting to C++, Java, Kotlin, Python or Rust, GPT-6.1 Sol scores 86.6% at low effort (GPT-5.6 Sol: 5.4%) and 97.9% at higher effort. All figures are OpenAI's own.

OpenAI open-sources MultiLangBench and reports low-resource language gains: GPT-6 Astra averages 87.4% building SQLite across 20 languages, GPT-6.1 Sol reaches up to 97.9% on COBOL migration

Key Takeaways

  • MultiLangBench: GPT-6 Astra (max) averages 87.4% across 20 languages vs 26.8% for GPT-5.6 Sol, above 83% in 19 of them (OpenAI-reported).
  • Per language: COBOL 10.9% to 72.1%, OCaml 25.6% to 85.8%, Fortran 21.4% to 83.5%, Visual Basic 27.5% to 89.4%.
  • Budget tier: GPT-6 Luna averages 40.3% at about $0.77 per sample vs $3.48 for GPT-5.6 Sol, roughly 78% cheaper.
  • COBOL migration: GPT-6.1 Sol scores 86.6% at low effort (GPT-5.6 Sol 5.4%) at about 61% of the API cost; 97.9% at higher effort, near saturation.
  • The eval setup, test suite and per-language results are on GitHub openai/multilangbench; the COBOL migration eval is not public.
Read full story