AI Coding Live News

Artificial Analysis Ships Endpoint Accuracy Index v1.1: Nearly 1 in 4 of 100+ Serverless Endpoints Score Below Reference

On 2026-11-06 Artificial Analysis launched Endpoint Accuracy Index v1.1: each serverless endpoint is scored as a percentage of a self-hosted official-weights reference across equally weighted BFCL-500, HLE-250, and AA-LCR-25. Vendor claim: nearly 1 in 4 of 100+ benchmarked endpoints score below the reference. AA-LCR-25 moved to v1.1 with repeats raised from 10 to 25 (average 95% CI ~±4.7 to ±2.6), and coverage expanded to six open-weights models including GLM-5.3 Flash and Muse Spark Open.

Artificial Analysis Ships Endpoint Accuracy Index v1.1: Nearly 1 in 4 of 100+ Serverless Endpoints Score Below Reference

Key Takeaways

  • Official: @ArtificialAnlys 2026-11-06 02:44 UTC; changelog 05 Nov 2026; methodology Endpoint Accuracy Index v1.1
  • Vendor claim: nearly 1 in 4 of 100+ serverless endpoints score below the self-hosted reference; providers often quantize, custom-kernel, tune stacks, or cap outputs
  • Equal-weight suite: BFCL v4-500 (3 repeats), HLE-250 (10), AA-LCR-25 (25); 95% CIs vs reference with significance flags
  • GLM-5.3 Flash: 16 of 22 endpoints statistically in line, one ~104.2; ~32K output caps scored ~22%/29% on HLE-250 vs 55% reference
  • Muse Spark Open: 3 of 5 endpoints ~99.5–99.9 (incl. FP4); two mainly lose on tool calling (BFCL-500 ~50%/51% vs 59%)
Read full story

NVIDIA open-sources Nemotron 3.5 Bolt E4B 4GB, a Gemma 4 E4B-based small model for local assistants, characters and robots with tool use

On 2026-10-23 (15:41 UTC) the official @NVIDIAAI account announced Nemotron 3.5 Bolt E4B 4GB, a small open model for local assistants, interactive characters and robots that can hold a conversation and use tools. The official Hugging Face model card says it is post-trained from Gemma-4-E4B-it, has 8B total parameters (4B effective), ships in Q4_K_M quantization, supports up to 64K input and 64K output tokens, and has a switchable thinking mode, plus a DFlash speculative-decoding drafter and a BF16 version. Weights, data and recipes are open under the OpenMDW-1.1 license, with commercial use allowed. On the model card the 4GB build scores 55.60 on MultiChallenge (Gemma4 E4B-it at the same quantization 39.36, Qwen3.5 4B 52.23), but trails Qwen3.5 4B on Tau2-Bench (52.32) and BFCL-v3 (67.31).

NVIDIA open-sources Nemotron 3.5 Bolt E4B 4GB, a Gemma 4 E4B-based small model for local assistants, characters and robots with tool use

Key Takeaways

  • NVIDIA officially released Nemotron 3.5 Bolt E4B 4GB for local assistants, interactive characters and robots
  • Post-trained from Gemma-4-E4B-it: 8B total parameters (4B effective), Q4_K_M quantization, 64K context, switchable thinking mode
  • Open weights, data and recipes under the OpenMDW-1.1 license with commercial use allowed, plus a DFlash speculative-decoding drafter
  • Model card results: 55.60 on MultiChallenge leads its size class, but it trails Qwen3.5 4B on Tau2-Bench and BFCL-v3
Read full story
Linked Models & Agents:
⚡ Nemotron 3.5 Bolt E4B⚡ Gemma 4 E4B🌐 NVIDIA🌐 Google

WEFT: Whole-System Evolution for Tool-Use Post-Training Scales Agentic Interaction with MegaMCP State Isolation

Recent attempts to scale tool-use post-training for autonomous agents have focused primarily on synthesizing standalone execution environments. However, environments constitute merely one component of a holistic agentic interaction system comprising environments, tasks, harnesses, and evaluators; scaling environments in isolation yields inconsistent training signals. Researchers from Fudan University introduce WEFT (arXiv:2609.36887), a whole-system evolution framework for tool-use post-training. WEFT coordinates environment breadth, task complexity, and interaction diversity, applying execution-driven self-evolution to attribute failures and iteratively revise flawed components. To ensure optimization stability and concurrency reliability, WEFT introduces prefix-preserving sampling, atomic-turn credit assignment, and MegaMCP—a Model Context Protocol infrastructure providing isolated, recoverable state management across concurrent rollouts. WEFT-8B and WEFT-14B outperform matched environment-scaling baselines across BFCL V4, tau^2-Bench, and Claw-Eval, with WEFT-14B beating Agent-World-14B by 6.41, 2.23, and 12.27 percentage points, while WEFT-35B-A3B excels on long-horizon benchmarks like Toolathlon-Verified.

WEFT: Whole-System Evolution for Tool-Use Post-Training Scales Agentic Interaction with MegaMCP State Isolation

Key Takeaways

  • Pioneers WEFT, co-evolving environments, tasks, harnesses, and evaluators to overcome the limits of isolated environment synthesis
  • Integrates prefix-preserving sampling, atomic-turn credit assignment, and MegaMCP isolation for resilient tool-use post-training
  • WEFT-14B surges past Agent-World-14B by 6.41 pp on BFCL V4, 2.23 pp on tau^2-Bench, and 12.27 pp on Claw-Eval
Read full story
Linked Models & Agents:

SLCA-GRPO: Segment-Locked Credit Assignment in Tool-Calling RL Resolves Reward Contamination

Addressing the structural vulnerability where standard GRPO broadcasts trajectory-level scalar advantages indiscriminately across heterogeneous tool calls and summary text, researchers introduce SLCA-GRPO. By establishing Segment-Locked Credit Assignment (SLCA), a Schema-Guided LLM Simulator (SGLS), and Hierarchical Rewards, the framework isolates execution advantages from preference noise without extra rollouts, driving a +9.15 pp gain on τ²-Bench and +1.36 pp on BFCL for 7B models.

SLCA-GRPO: Segment-Locked Credit Assignment in Tool-Calling RL Resolves Reward Contamination

Key Takeaways

  • Eliminating CSCM Contamination: Fixes cross-segment credit misattribution (CSCM) where GRPO indiscriminately assigns trajectory-level advantages across heterogeneous text and tool segments within a single rollout group.
  • Hierarchical Rewards & Zero-API Exploration: Introduces the Schema-Guided LLM Simulator (SGLS) and Hierarchical Rewards (HierR) to direct execution advantages to tool tokens and conversational preferences to summary tokens.
  • Comprehensive Benchmark Gains: On a 7B backbone, outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp in-domain, +1.36 pp on BFCL, and +9.15 pp on τ²-Bench while cutting tool redundancy and overhead.
Read full story
Linked Models & Agents:
ADSponsored