On 2026-11-06, Scope (YC P26) founder Anand Paj (@anandPa94) released the first hosted-database agent signup benchmark. Agents start with no account, provision the named DB, write a record, and hand back credentials across 20 Claude Code/Codex/Cursor configs and 1,035 sessions. Official leaderboard: Prisma Postgres ~88.3%, Upstash ~63.3%, Neon/Nile ~43.3%; CAPTCHAs stopped 373 sessions; 7 of 18 products had zero successes.
Zoom
Key Takeaways
Source: Scope founder Anand Paj (@anandPa94) X post 2026-11-06 + official board tryscope.com/leaderboard/databases (vendor/evaluator)
Scale: 18 hosted DBs × 20 agent configs = 1,035 sessions; only ~16.6% reached a first working result overall
On 2026-11-05, Hugging Face Harness Optimization researcher Joël Niklaus (@joelniklaus) released the Harness Leaderboard: same model, provider, and reasoning effort with only the harness swapped, covering GLM-5.3-Flash, DeepSeek-V4.1-Flash, and Flint on Terminal-Bench 2.1 and DeepSWE—28,849 trials and about $4,746 of inference. Author-reported averages: mini-swe-agent ~51.4% resolve rate; Pi ~49.4% at about $0.29 per solved task. This is an individual research leaderboard, not a Hugging Face company product announcement.
Zoom
Key Takeaways
Source: personal post by HF Harness Optimization researcher Joël Niklaus on 2026-11-05T16:04:08Z—not a company announcement
Scale (author): 9 harnesses on open models, 28,849 trials, ~$4,746 inference spend
Method: pin model/provider/reasoning effort; 3 attempts per task; Terminal-Bench 2.1 and DeepSWE
Results (author): averaged mini-swe-agent ~51.4%, openclaw ~27.3%; Pi ~49.4% at ~$0.29 per solved task
Use: public HF Space leaderboard + trajectories; Harness Lens documents prompts/tools for each harness
On November 5 Julius Research CEO rahul (@0interestrates; Julius AI) released DeepFinance: 21 ultra long-horizon PE/IB tasks built from anonymized real deal data rooms (~20–30 human hours each, mean ~27.1h; ~84 rubric criteria). Official leaderboard: Opus 5.5 (max) ~54.8%, Sonnet 5.5 ~54.6%, GPT-6 Astra ~54.2%. Focus is auditable trajectories, not final-number checks alone.
Zoom
Key Takeaways
Primary source: Julius Research CEO @0interestrates post + official leaderboard/blog (first-hand lab lead release, not a separate corporate PR account)
Scale: 21 tasks; mean ~27.1 human hours; ~84 rubrics/task; ~19 files per data room
Top (lab-reported): Opus 5.5 max ~54.8%; Sonnet 5.5 ~54.6%; GPT-6 Astra xhigh ~54.2%
Cost (official table): Opus ~$8.63/task; Astra ~$4.55; GPT-6.1 Sol ~$0.81 at ~52.9%
Gap filled: moves beyond doc QA / final-number checks toward auditable trajectories under conflicting evidence
On Nov 2, semiconductor and AI research firm SemiAnalysis released GDPx, a benchmark of agentic capability across 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement (RSI), hardware design, materials science, alignment and software engineering. Its 206 tasks were built by 14 specialist data partners, with SemiAnalysis doing final QA. On the official leaderboard Claude Opus 5.5 leads at 12.0%, followed by Gemini 4 Carbon at 10.9% and GPT-6 Astra at 9.6%; nearly every model scores 0 on finance and legal. SemiAnalysis says safety classifiers drag down Opus 5.5 and GPT-6 Astra, and plans a new version every 3–6 months. All scores are as reported by SemiAnalysis.
Zoom
Key Takeaways
Scale: 11 domains, 14 data partners and 206 tasks, with a default time limit of 8 hours per task (GDPx site)
Leaderboard (as reported by SemiAnalysis): Claude Opus 5.5 12.0%, Gemini 4 Carbon 10.9%, GPT-6 Astra 9.6%, Muse Wonder Full 9.0%, Claude Fable 5.1 8.5%, GPT 6.1 Sol 8.4%, Claude Sonnet 5.5 8.2%
Lower tier and open weights: DeepSeek 4.1 Flash 1017 5.4%, GPT-6 Luna 5.1%, Grok 4.7 3.9%, GLM 5.3 2.7%; almost every model scores 0 on finance and legal
Method: each model runs in its native harness where one exists (Claude Code for Opus, Codex for Astra, Antigravity for Carbon), otherwise OpenCode; the headline is pass@1 over three attempts with all domains weighted equally (SemiAnalysis blog)
By domain: Opus 5.5 wins quant trading, hardware and chip design; Gemini 4 Carbon wins biology and software engineering; GPT-6 Astra wins RSI and materials science; Muse Wonder Full is the strongest cyber model
Kaggle announced ScarfBench with IBM Research, a 204-task benchmark that tests whether coding agents can migrate enterprise Java apps between Jakarta EE, Quarkus and Spring without changing behavior. IBM's October 27 technical note shows Gemini 4 Argon with Antigravity first on the smoke score at 75.2%, Claude Opus 5.5 with Claude Code highest on all-tests-pass at 72.1%, and GPT-6.1 Sol with Codex cheapest at $1.25 per solved task.
Zoom
Key Takeaways
204 migration tasks and 10 model-plus-harness configurations; top smoke scores are Gemini 4 Argon + Antigravity 75.2%, Claude Opus 5.5 + Claude Code 70.5% and Claude Opus 5 + Claude Code 69.8% (IBM figures)
Claude Opus 5.5 + Claude Code has the highest all-tests-pass rate at 72.1%, just ahead of Gemini 4 Argon at 71.1%
Compiling is mostly solved: 8 of 10 configurations compiled at least 97% of solvable tasks, yet smoke scores spanned more than 50 points
535 of 745 classified failing runs (about 72%) were runtime behavioral failures, including 210 lost-route/404 failures
Cost per solved task: $1.25 for GPT-6.1 Sol + Codex, $1.59 for Gemini 3.8 Flash + mini-swe and $10.10 for Claude Opus 5.5 + Claude Code; public submissions are not open yet
Cognition, the company behind Devin, posted on X at 01:45 UTC on Oct 30 that Google's new Gemini 4 Carbon scores 40% on its FrontierCode 2.0 coding benchmark at $4.04 per task, near the top and behind only Claude Opus 5.5, GPT-6 Astra and Claude Sonnet 5.5. The earlier Gemini 4 Argon scores 26.3% at a higher $6.41 per task. The results are Cognition's own runs, and the leaderboard on its site has been updated.
Zoom
Key Takeaways
Gemini 4 Carbon scores 40% on FrontierCode 2.0 at $4.04 per task, behind Claude Opus 5.5 (45.5%), GPT-6 Astra (42.8%) and Claude Sonnet 5.5 (post)
Gemini 4 Argon scores 26.3% at $6.41 per task, ahead of Muse Wonder 1.0 but behind Claude Haiku 5.5 and Grok 4.7
From Argon to Carbon the score rises about 13.7 points while cost per task drops about $2.37
The changelog shows Gemini 4 Carbon was added on Oct 28; FrontierCode 2.0 has 132 tasks scored on whether a maintainer would merge the PR (leaderboard)
All figures are Cognition's own runs; Google has not published FrontierCode results itself
NVIDIA AI announced Nemotron 3.5 Transcribe on Oct 28, with weights on Hugging Face and ModelScope. It is an offline, instruction-driven speech model whose single checkpoint covers transcription in 40 languages, translation between 31 languages and English, speaker attribution in 13 languages, and audio question answering, on recordings up to an hour long with up to eight labeled speakers. NVIDIA says it ranks first among open-weight models on the Hugging Face Open ASR English leaderboard at 5.33% average WER, and Artificial Analysis measured a 2.8% AA-WER, tied for first among open models with Voxtral Small, which uses 24B active parameters. The weights, the Granary v2 training data and fine-tuning recipes are all released.
Zoom
Key Takeaways
Architecture: a speech encoder paired with a Mamba-2/Transformer hybrid MoE language model, about 30B total parameters (34B in the model card counting the audio encoder) and about 4B active per token (4.3B in the model card), shipped in BF16, FP8 and NVFP4
Capabilities: transcription in 40 languages, translation between 31 languages and English, speaker attribution in 13 languages, audio Q&A (trained in English only) and word-level timestamps, on recordings up to 1 hour with up to 8 speakers
Official and third-party figures: first among open models on the Open ASR English leaderboard (5.33% average WER), 4.62% mean WER across 7 languages and 10.98% on long-form; Artificial Analysis AA-WER 2.8% and 1.8% on its AA-AgentTalk voice-agent set (next-best open model 2.3%)
Artificial Analysis measured DeepInfra's hosted endpoint at a 75x speed factor, transcribing a minute of audio in under a second, priced at $2.00 per 1,000 minutes
Licensed under OpenMDW-1.1 for commercial use; Granary v2 data and NeMo Speech fine-tuning tools are released too, and in NVIDIA's medical case study fine-tuning cut WER a further 45.8% relative to the best prompt
Microsoft AI announced on Oct 27 that MAI-Image-2.6 and MAI-Image-2.6-Flash are generally available in Microsoft Foundry, with multi-reference editing from up to five images, Bing web grounding, output up to 1.5K and published per-token pricing. The same day its next image editing model, MAI-Image-2.7-Preview, reached #3 overall in Arena's Image Edit leaderboard at 1470 points, 45 points above 2.6; for now it is available only for editing in MAI Playground.
Zoom
Key Takeaways
MAI-Image-2.6 and MAI-Image-2.6-Flash are generally available in Microsoft Foundry, and Microsoft says 2.6 is coming soon to Adobe Firefly
Official pricing: 2.6 costs $5 text input, $8 image input and $38 image output per 1M tokens; Flash costs $1.75, $2.50 and $19
2.6 supports multi-reference editing with up to five images, Bing web grounding, dynamic aspect ratios and output up to 1.5K
Per Arena, MAI-Image-2.7-Preview is #3 overall in Image Edit Arena at 1470 points, 45 points above 2.6 (#7), putting Microsoft AI #2 among labs
Per Artificial Analysis, 2.7-Preview is #2 on AA-Image-Editing v2.1 and beats 2.6 on all 7 editing actions, ranking #1 on 4 of them
Cursor announced on Oct 26 (UTC) that DeepSeek V4.1 Flash is now available in Cursor and can be turned on under Cursor Settings > Models. Cursor says every effort level sits on the CursorBench cost frontier, from $0.60 to $0.98 per task, with no model scoring higher at those prices. The official CursorBench 4.0 leaderboard lists DeepSeek V4.1 Flash 1017 at 47.7% on Max ($0.98 per task), 46.3% on Medium ($0.80), 45.6% on High ($0.77) and 44.9% on Low ($0.60). The top score on the board, Opus 5.5 Max, is 57.8% but costs $13.43 per task.
Zoom
Key Takeaways
Four effort levels on the official CursorBench 4.0 board: Max 47.7% at $0.98, Medium 46.3% at $0.80, High 45.6% at $0.77, Low 44.9% at $0.60; Max ranks 17th overall
Value: Max at 47.7% nearly ties Sonnet 5.5 High (47.8%, $1.20) for about 18% less; Opus 5.5 Max scores about 10 points higher but costs $13.43 per task, roughly 13.7x more
Nearest rivals on price: Haiku 5.5 Max at 48.4% for $1.12 and Haiku 5.5 Extra High at 44.3% for $0.56, bracketing DeepSeek's price range
The trade-off is tokens and steps: Max averages about 537K tokens and 187 steps per task, versus about 37K tokens and 41 steps for Sonnet 5.5 High, so expect longer runs
Turn it on under Cursor Settings > Models; a counterintuitive detail is that Medium (46.3%) scores slightly above High (45.6%) for only $0.03 more
On Oct 22 Canva released Canva Craft, the first image-generation foundation model trained from scratch by its in-house lab, Canva Research. It works in two stages: a planning model writes out the composition (elements, spatial relationships, hierarchy and any text), then a rendering model produces the image at up to 2K across more than 30 aspect ratios. Training leaned heavily on millions of human-made templates from Canva's library, such as posters, social posts, slides and infographics. Canva says it debuted 4th overall on the crowdsourced DesignArena leaderboard and 1st in the People & Portraits and Landscapes & Nature categories. It is available now to all Canva users in Dream Lab and to developers through the Connect API as craft_v1_2k, with Canva MCP access coming soon.
Zoom
Key Takeaways
Two-stage design: a planning model writes the composition (elements, relative scale, hierarchy, text), then a rendering model outputs up to 2K across 30+ aspect ratios (announcement)
Vendor-reported ranking: 4th overall on DesignArena, 1st in People & Portraits (of 87 models) and Landscapes & Nature (of 85), 2nd in Abstract & Patterns and 4th in Products
Training emphasized millions of human-made Canva templates; designers judged more than 25,000 head-to-head image pairs against leading models
For developers: POST /v1/image-generations on the Connect API with model set to craft_v1_2k, which spends the user's AI credits (API docs)
Prompts work in any language and in-image text covers 55 languages; the faster, cheaper Craft-fast and Craft-ultrafast variants and MCP access are not out yet
On Oct 20, Krea said Krea 3, the image-editing model it released on Oct 15, has entered several third-party leaderboards. On Arena's Multi-Image Edit board, Krea 3 Ultra ranks #4 at 1435±11 (preliminary, 2,991 votes), behind three OpenAI models; it is #8 on single-image editing. On Artificial Analysis, Ultra scores 1151 Elo (#5) and Fast scores 1125 (#11). Krea 3 edits at native 2K, returns most edits in under 8 seconds and costs $0.06 per image.
Zoom
Key Takeaways
Arena Multi-Image Edit: Krea 3 Ultra is #4 at 1435±11 on 2,991 votes, marked preliminary; the top 3 are two GPT-Image-2.5 variants and GPT-Image-2
Arena single-image Image Edit: Krea 3 Ultra is #8 at 1411±11 on 2,619 votes
Artificial Analysis Image Editing v2.1: Ultra is #5 at 1151 Elo (statistically tied for #3 with Nano Banana 2.1 and MAI-Image-2.6); Fast is #11 at 1125
Design Arena Image Editing: #4 of 51 models at 1298 Elo
Vendor figures: most edits under 8 seconds, native 2K, $0.06 per image, up to 6 reference images
Independent AI evaluation authority Artificial Analysis launched the HealthBench and HealthBench Professional clinical reasoning and medical conversation leaderboards. Addressing the critical flaw where models artificially inflate medical scores through exhaustive, padded responses, the Professional benchmark implements rigorous length penalties. GPT-6 Astra (max effort) leads the world at 66.0%, with GPT-6.1 Sol at 65.1% and Claude Sonnet 5.5 at 63.0%. Claude Opus 5.5, despite holding a 71.4% raw score, collapses to 59.9% (#6) due to averaging ~5,900 characters per reply.
Zoom
Key Takeaways
Artificial Analysis launched HealthBench, introducing strict length penalties on the Professional leaderboard to penalize verbose medical answers
GPT-6 Astra took #1 with 66.0%, followed by GPT-6.1 Sol at 65.1% and Claude Sonnet 5.5 at 63.0%
Claude Opus 5.5 scored 71.4% raw, but dropped to 59.9% (#6) under length penalties due to averaging ~5,900 characters per reply
On standard HealthBench, DeepSeek V4.1 Flash took #2 with 60.9%, outperforming multiple frontier commercial models
On October 19 independent evaluator Artificial Analysis launched two health-conversation leaderboards built on OpenAI's HealthBench (5,000 realistic health conversations) and HealthBench Professional (525 clinician conversations), graded against physician-written rubrics and reported as both raw and OpenAI-style length-adjusted scores. On Professional, GPT-6 Astra (max) leads at 66.0% length-adjusted, ahead of GPT-6.1 Sol (65.1%) and Claude Sonnet 5.5 (63.0%); Claude Opus 5.5 has the top raw score (71.4%) but drops to sixth (59.9%) because its ~5,900-character answers are penalized. On HealthBench, Claude Sonnet 5.5 leads at 62.1%, with DeepSeek V4.1 Flash close behind at 60.9% for about 1/16 of the cost per task.
Zoom
Key Takeaways
HealthBench Professional (length-adjusted): GPT-6 Astra (max) 66.0%, GPT-6.1 Sol (max) 65.1%, Claude Sonnet 5.5 63.0%.
Claude Opus 5.5 posts the highest raw score (71.4%) but averages ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and falls to sixth at 59.9%; Claude Fable 5.1 also loses over 11 points.
The Professional score-vs-cost Pareto frontier is all OpenAI: GPT-6 Luna (max) scores 61.9% at $0.004 per task, under 2% of GPT-6 Astra's $0.27.
HealthBench: Claude Sonnet 5.5 leads at 62.1%, DeepSeek V4.1 Flash 60.9% at $0.005 per task, GPT-6.1 Sol 58.5%.
Grading follows OpenAI's original setup: GPT-5.4 (low) grades Professional and GPT-4.1 grades HealthBench.
MiniMax officially announced M3.1 Flash, its next-generation lightweight frontier model (previously dominating leaderboards as the secret model Space Bunny across 200T+ tokens). Featuring a native 1M context window, flat-rate API pricing of $0.15 / $0.60 per 1M input/output tokens ($0.015 cache reads), and 3x faster inference at 1/3 the cost of M3. It surged +106 points to 1588 on LMSYS Code Arena: WebDev (#12 among open models), with full weights set for open release on October 23.
Zoom
Key Takeaways
MiniMax officially launched M3.1 Flash, unveiling that it was the mystery model Space Bunny evaluated across 200T+ tokens
Features native 1M context with flat-rate API pricing of $0.15/$0.60 per 1M tokens ($0.015 cache reads), offering 3x speed at 1/3 cost of M3
Debuts on LMSYS Code Arena: WebDev at 1588 Elo (+106 pts over M3), ranking #12 among open models with major leaps in Gaming (+179) and Reference Design (+155)
Artificial Analysis benchmark confirms Intelligence Index of 42 and HLE of 48% at half the token consumption of rivals; weights release on Oct 23
Google Labs launched Playground on 2026-10-07: describe a game in chat, test it, and share it or publish it to the Explore gallery, with multiplayer and leaderboards in select genres. It runs on Gemini, Nano Banana, and Lyria with a custom harness. US 18+ first; weekly creation tokens with higher limits on Google AI plans; Unity Spark integration coming.
Zoom
Key Takeaways
Official: Google Labs launched the Playground experiment on 2026-10-07; no-code, prompt-based game creation
Availability: US, 18+, at playground.google; weekly creation tokens, higher on Google AI subscriptions
Stack: Gemini + Nano Banana + Lyria with a custom harness (per Google spokesperson to The Verge)
Sharing: private, link, or Explore gallery; multiplayer and leaderboards in select genres; safety screening
Unity Spark integration coming (in testing, closed beta soon); no published quality or latency metrics
