AI Coding Live News

Artificial Analysis Ships Endpoint Accuracy Index v1.1: Nearly 1 in 4 of 100+ Serverless Endpoints Score Below Reference

On 2026-11-06 Artificial Analysis launched Endpoint Accuracy Index v1.1: each serverless endpoint is scored as a percentage of a self-hosted official-weights reference across equally weighted BFCL-500, HLE-250, and AA-LCR-25. Vendor claim: nearly 1 in 4 of 100+ benchmarked endpoints score below the reference. AA-LCR-25 moved to v1.1 with repeats raised from 10 to 25 (average 95% CI ~±4.7 to ±2.6), and coverage expanded to six open-weights models including GLM-5.3 Flash and Muse Spark Open.

Artificial Analysis Ships Endpoint Accuracy Index v1.1: Nearly 1 in 4 of 100+ Serverless Endpoints Score Below Reference

Key Takeaways

  • Official: @ArtificialAnlys 2026-11-06 02:44 UTC; changelog 05 Nov 2026; methodology Endpoint Accuracy Index v1.1
  • Vendor claim: nearly 1 in 4 of 100+ serverless endpoints score below the self-hosted reference; providers often quantize, custom-kernel, tune stacks, or cap outputs
  • Equal-weight suite: BFCL v4-500 (3 repeats), HLE-250 (10), AA-LCR-25 (25); 95% CIs vs reference with significance flags
  • GLM-5.3 Flash: 16 of 22 endpoints statistically in line, one ~104.2; ~32K output caps scored ~22%/29% on HLE-250 vs 55% reference
  • Muse Spark Open: 3 of 5 endpoints ~99.5–99.9 (incl. FP4); two mainly lose on tool calling (BFCL-500 ~50%/51% vs 59%)
Read full story

Artificial Analysis Makes Time per Task Its Headline Performance Metric: GPT-6 Astra Finishes a Max-Effort Task in About 17 Minutes, Ahead of Opus 5.5 and Gemini 4 Carbon

Independent benchmarker Artificial Analysis announced on X on November 4 (17:34 UTC) a new version of its API performance benchmarking: the headline metric moves from output speed to Time per Task, the time to finish one Intelligence Index task (output tokens per task divided by end-to-end output speed). It covers about 1,000 serverless endpoints, measures speed from sending the request to the final token, and adds multi-turn agentic coding and knowledge-work workloads. Per its data, GPT-6 Astra at max effort runs at 51 tokens/s yet finishes a task in about 17 minutes, against about 26 for Claude Opus 5.5 (max) and about 29 for Gemini 4 Carbon (high).

Key Takeaways

  • Time per Task equals output tokens per Intelligence Index task divided by end-to-end output speed, so a model that is fast per token but verbose can finish later than a slower model that uses fewer tokens.
  • End-to-end output speed runs from sending the request to the final token, including the wait before the first token (prefill and hidden reasoning); output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer.
  • GPT-6 Astra (max) runs at 51 tokens/s with about 50k output tokens per task and takes about 17 minutes; Claude Opus 5.5 (max, 111 tokens/s) takes about 26 minutes and Gemini 4 Carbon (high, 71 tokens/s) about 29, using roughly 120k to 170k tokens per task.
  • At each lab's API default effort (medium for all three), GPT-6 Astra, GPT-6.1 Sol and Opus 5.5 cut Time per Task by about 3 to 5x and Cost per Task by about 2 to 4x, scoring about 5 to 11 points lower; Opus 5.5 drops from about 26 minutes and $10.04 to about 6.8 minutes and $2.45.
  • The official methodology page lists this as version 3.0.0 (October 30), with runs now every 6 hours and a new endpoint error rate metric.
Read full story

Fireworks Forge Launches: Ultra-fast Inference Preview Hits 1,500 Tokens/s, Half-Price Background Mode, Nexus Desktop for Codex and Claude Desktop, Sovereign Private Preview

At its first Forge conference on November 3, open-model inference platform Fireworks launched a batch of products. Ultra-fast, powered by NVIDIA Blackwell, is in public preview at up to 1,500 tokens per second on DeepSeek V4.1 Flash, cutting a 30-turn agent task from 1 minute to 8 seconds. Background Mode runs deferrable work at half price, alongside multi-day prompt caching and higher rate limits. Nexus Desktop brings open-model routing into the Codex app, Claude Desktop and OpenCode, with Pro plans from $8 per month. Fireworks also announced No Eval, serverless LoRA inference, the Fireworks Sovereign private preview and a 60+ partner Frontier Ecosystem.

Fireworks Forge Launches: Ultra-fast Inference Preview Hits 1,500 Tokens/s, Half-Price Background Mode, Nexus Desktop for Codex and Claude Desktop, Sovereign Private Preview

Key Takeaways

  • Ultra-fast public preview: on NVIDIA Blackwell, up to 1,500 tokens per second on DeepSeek V4.1 Flash; in Fireworks' benchmark a 30-turn agent task fell from 1 minute to 8 seconds (Fireworks blog)
  • Cost options: Background Mode runs deferrable work at half price; multi-day prompt caching cuts the cost of resuming long context; higher serverless rate limits and Elastic Burst are generally available
  • Nexus Desktop launches: personalized routing across open and closed models inside the Codex app, Claude Desktop and OpenCode; Pro from $8 per month, Teams from $10 per seat per month
  • Training: No Eval (private preview) turns production traces into task evals; serverless LoRA inference (public preview) starts with GLM 5.3 Flash at per-token pricing with no idle charge; free training on Muse Glimmer 30B through November 30
  • Enterprise: Fireworks Sovereign (private preview) runs the inference engine in a customer's own cloud or data center; Fireworks says it already manages over 25,000 GPUs in customer environments; plus a 60+ partner Frontier Ecosystem
Read full story

Fireworks ships DeepSeek V4 Flash with native vision at $0.22/$0.66 per 1M tokens

Fireworks said DeepSeek V4 Flash now includes native vision on serverless and priority tiers. It reports better benchmarks than the text-only 0731 checkpoint at the same $0.22 input / $0.66 output per 1M tokens. The model page lists a 305B MoE with about 1040K context, function calling, image input, and a speculative-decoding module.

Key Takeaways

  • DeepSeek-V4-Flash-Vision-Exp is live on Fireworks serverless and priority tiers with native image input.
  • Same price as text-only 0731: $0.22 input / $0.66 output per 1M tokens (cached input $0.007).
  • 305B MoE, ~1040K context, function calling, and speculative decoding for screenshot-aware coding agents.
Read full story
ADSponsored

Fireworks opens Training API and Fireworks Lab for serverless or dedicated RL at scale

Fireworks announced general availability of its Training API and Fireworks Lab, aiming to make model specialization accessible beyond labs that already own training clusters. The API lets teams connect a custom Python training loop to Fireworks-managed distributed training and rollout infrastructure, keeping control of loss, reward, data, and environment. Compute is split into Serverless mode for fast experiments and Dedicated mode when scale, control, and GPU economics matter. The stack is built for reinforcement learning as a single train-sample-serve-retrain loop, with three hard requirements: rollout throughput, weight sync so fresh weights land on the rollout deployment every step, and correctness so trainer and inference engine agree on the same model. Fireworks Lab additionally embeds forward-deployed researchers and engineers to co-design or build a model. Named customers include Cursor, Harvey, Figma, Vercel, Cognition, Factory, and Heidi.

Key Takeaways

  • The Training API attaches a custom Python loop to managed distributed training and rollout, with Serverless and Dedicated compute.
  • The RL loop is optimized for rollout throughput, per-step weight sync, and trainer-inference model agreement.
  • Fireworks Lab embeds researchers with customers including Cursor, Harvey, Figma, Vercel, Cognition, and Factory.
Read full story

Replit Agent partners with Neon for instant database branching and isolated sandbox data mutations

Replit partnered with Neon to bring instant serverless Postgres database branching directly into Replit Agent. When the agent tests schema migrations or executes destructive seed operations, it automatically spawns an ephemeral Postgres branch in 500ms, safeguarding production data until changes are verified.

Key Takeaways

  • Spawns an isolated serverless Postgres database branch in under 500ms.
  • Destructive migrations and seed scripts run safely in isolated sandbox branches.
  • Atomic merge back to main branch once automated tests pass successfully.
Read full story
Linked Models & Agents:

Fireworks AI Delivers Day-Zero US-Hosted Serverless Inference for GLM-5.3

Fireworks AI launched day-zero US-hosted serverless inference for GLM-5.3, highlighting a 50% improvement on Code Bench and state-of-the-art results in open-weight cyber defense and long-horizon agents.

Key Takeaways

  • Instant day-zero serverless API access for teams without dedicated multi-GPU infrastructure for 744B MoEs;
  • Optimized specifically for complex code refactoring, cyber security benchmarks, and long-horizon tasks;
  • Proprietary inference stack delivers sustained high tokens-per-second streaming performance.
Read full story
Linked Models & Agents:

Together AI Launches Hosted GLM-5.3 Flash API: 320B Multimodal MoE with 2x Output per Budget on DeepSWE

Together AI launched hosted serverless API serving for GLM-5.3 Flash, highlighting its 320B-A18B multimodal MoE architecture, 1M context, and high DeepSWE efficiency compared to proprietary frontier models.

Key Takeaways

  • Brings 320B total / 18B active multimodal MoE into hosted serverless APIs with instant elastic scaling;
  • Native visual reasoning combined with hybrid attention reduces input token consumption for agentic workflows;
  • Matches frontier-level engineering accuracy on DeepSWE while delivering more than double the completed tasks per dollar.
Read full story
Linked Models & Agents:

Modal Labs Rolls Out GLM-5.3-Flash Auto Endpoints for Serverless GPU Inference

Modal Labs announced Auto Endpoints for GLM-5.3-Flash, enabling developers to deploy and query the 320B-A18B MoE model via OpenAI-compatible endpoints with automated serverless cold-start management.

Key Takeaways

  • Day-0 open-model availability establishes the baseline competitive standard for serverless GPU platforms;
  • Auto Endpoints abstract away containerization and GPU orchestration into immediate OpenAI-compatible calls;
  • Elastic on-demand scaling eliminates idle infrastructure costs for bursty multi-turn coding agent workloads.
Read full story
Linked Models & Agents: