AICoder

AI Coding Live News

First-hand AI coding updates from 𝕏/Twitter frontier labs and researchers, distilled by AI with entity benchmarks.

Explore live on 𝕏 ↗

AI Coding Live News

v
vLLM Project@vllm_project·
👀62
🔥92
View on 𝕏 ↗

vLLM Ships Precompiled Bytecode Images and Decision 3.0 Serving: Cold Start Accelerated by 18%, MI325X Decision Throughput Hits 1.8x

The vLLM team announced two major milestones: shipping precompiled Python bytecode via uv in v0.32.0 release images cuts cold-start container launch by 14–18% (~18s on NVIDIA H100) with zero runtime recompiles. Additionally, vLLM integrated the Decision 3.0 (D3) structured reasoning family (0.6B to 27B) into the /v1/systemone endpoint, ranking #1 on the Jev Decision Index and delivering 1.8x req/s on a single AMD MI325X under 64 concurrent clients.

vLLM Ships Precompiled Bytecode Images and Decision 3.0 Serving: Cold Start Accelerated by 18%, MI325X Decision Throughput Hits 1.8x

Key Takeaways

  • vLLM ships precompiled Python bytecode via uv in official images, reducing H100 cold starts by 14–18% (~18 seconds saved)
  • Natively serves the Decision 3.0 (D3) family (0.6B to 27B) via the /v1/systemone structured decision endpoint
  • D3 models rank #1 on the Jev Decision Index with 66.3 in text and 71.3 in vision
  • D3 27B achieves 1.8x the request throughput of standard checkpoint servers on a single AMD Instinct MI325X under 64 concurrent clients
Read full story
Linked Models & Agents:
#vLLM#Cold Start#Decision 3.0#AMD MI325X#System One#Inference
v
vLLM@vllm_project·
👀176
🔥88
View on 𝕏 ↗

vLLM v0.32.0: 814 commits, built-in /v1/systemone structured-decision endpoint, NVFP4 KV cache on more GPUs, and 43–85% more req/s with multiple API workers

vLLM shipped v0.32.0 on Oct 18 with 814 commits from 363 contributors (121 new). Highlights: a default-served POST /v1/systemone structured-decision endpoint and a generic hf tool/reasoning parser; NVFP4 KV cache on SM8x/SM90/SM12x and FP8 KV by default for GLM-5.x on SM100; Model Runner V2 gains a GPU n-gram speculator and Elastic EP and becomes the ROCm default; three security advisories are fixed, default CORS is restricted, and Python 3.10 is dropped.

vLLM v0.32.0: 814 commits, built-in /v1/systemone structured-decision endpoint, NVFP4 KV cache on more GPUs, and 43–85% more req/s with multiple API workers

Key Takeaways

  • Scale: 814 commits from 363 contributors (121 new); see the release notes.
  • Structured decisions: POST /v1/systemone is new and served by default for choice/score questions, including Cloudflare/clef-omni and LiquidAI/d1-3B; --tool-call-parser hf reads a checkpoint's response_template so no model-specific parser is needed.
  • Low-precision KV: GLM-5.x defaults to FP8 KV on SM100, halving KV memory per rank with 2.3–5.5% more throughput at a 7–11% TTFT cost (PR benchmark, vendor figures); ROCm adds a 4-bit KV cache at about 2x FP8 density.
  • Serving: with --api-server-count above 1, connections go to the least-loaded worker for 43–85% more req/s on short requests (Qwen3-0.6B, 32 in / 1 out tokens, PR benchmark).
  • Breaking: Python 3.10 dropped, default CORS origins restricted, cascade attention removed, and max_tokens is no longer reserved from the context window.
Read full story
Linked Models & Agents:
⚡ GLM-5.3⚡ DeepSeek-V4.1-Flash⚡ Kimi K3⚡ MiniMax-M3⚡ qwen3.8-flash-next🌐 vllm
#vLLM#v0.32.0#Inference Engine#Systemone#NVFP4#KV cache#Model Runner V2#Security
X
Xuan-Son Nguyen / ggml-org@ngxson·
👀184
🔥96

llama.cpp ships Decision Models: /v1/systemone scores options in one forward pass (Julia-1, Laya, Kev-4B, lev, OpenJev)

On 2026-10-02 ggml-org announced llama.cpp server support for Decision Models via /v1/systemone (PR #29818, merged 2026-10-02T09:56Z). Send a state plus typed questions; get option probabilities in one forward pass. Day-0 models: Julia-1, Laya, Kev-4B, lev, OpenJev (vision). Cloudflare Clef is planned next.

llama.cpp ships Decision Models: /v1/systemone scores options in one forward pass (Julia-1, Laya, Kev-4B, lev, OpenJev)

Key Takeaways

  • Official post 2026-10-02 + merged PR #29818 (2026-10-02T09:56Z, +2139/−15)
  • Endpoint POST /v1/systemone with choice/score/noul; probabilities + confidence; output_tokens=0
  • Day-0 GGUFs: Julia-1 / Laya / Kev-4B / lev / OpenJev; median ~3–43 ms/question on RTX PRO 6000 (blog table)
  • Distinct from Ollama SystemOne & Cloudflare Clef hosting; Clef listed as next in llama.cpp
  • Start: llama serve -hf ggml-org/Kev-4B-GGUF then curl /v1/systemone
Read full story
Linked Models & Agents:
⚡ Julia 1⚡ laya⚡ Kev 4b⚡ lev⚡ openjev🌐 ggml-org🌐 huggingface
#llama.cpp#Decision Models#System One#Systemone#Jev#GGUF#local inference#agent routing#Kev-4B#OpenJev#Ggml
O
Ollama@ollama·
👀74
🔥90

Ollama Python 0.6.3: native systemone() for Decision Models; think accepts model-defined levels

Official Ollama Python client v0.6.3 shipped 2026-09-29: adds ollama.systemone() matching server v0.35.0 /v1/systemone Decision Models (choice/noul/score); relaxes think to string levels; fixes multi-tool examples and case-insensitive image extensions. Requires Ollama ≥0.35.0 plus local decision models such as nimble/tev1.

Ollama Python 0.6.3: native systemone() for Decision Models; think accepts model-defined levels

Key Takeaways

  • Release v0.6.3 on 2026-09-29; ~6 commits vs v0.6.2 including examples/systemone.py
  • New ollama.systemone(model, state, questions) batches choice/noul/score Decision Model queries
  • Needs Ollama server v0.35.0+ and a local decision model (nimble/tev1)
  • think accepts string levels for model-defined thinking (#697/#744)
  • Install: pip install -U ollama==0.6.3
Read full story
Linked Models & Agents:
⚡ nimble⚡ tev1🤖 ollama🌐 Ollama
#Ollama#Python SDK#Decision models#Systemone#Release
ADSponsored
O
Ollama@ollama·
👀137
🔥93

Ollama 0.35.0: Decision Models via /v1/systemone (Jev); Nimble & Tev1 return choices, probabilities, confidence

Ollama shipped v0.35.0 on 2026-09-28 (GitHub prerelease): Decision Models via TypeSafe’s Jev API at /v1/systemone. Instead of free text, models answer choice/noul/score questions with options, probabilities, and confidence—for ticket triage, model routing, and classification. First models: Nimble (Bespoke Labs) and Tev1 (Together AI). ~7 commits / 66 files vs v0.34.4; Settings no longer blocks on discovery; macOS update menu and stalled MLX downloads fixed.

Ollama 0.35.0: Decision Models via /v1/systemone (Jev); Nimble & Tev1 return choices, probabilities, confidence

Key Takeaways

  • v0.35.0 (prerelease) adds Decision Models at POST /v1/systemone (TypeSafe Jev)
  • Question types: choice (with probabilities), noul, score — not free-text generation
  • First models: Nimble (Bespoke Labs) and Tev1 (Together AI) on ollama.com/library
  • ~7 commits / 66 files vs v0.34.4; Settings/macOS update/MLX download fixes
Read full story
Linked Models & Agents:
⚡ Nimble⚡ Tev1⚡ Jev🤖 Decision Models🤖 model routing🌐 ollama🌐 TypeSafe🌐 Bespoke Labs🌐 Together AI
#Ollama#Decision Models#Systemone#Jev#Nimble#Tev1#Classification#Routing
U
Unsloth@unslothai·
👀102
🔥94

Unsloth v0.1.900-beta: local Laya Decision models + Skills editor; ~4.5× LTX-2.3 clips and up to 6.3× faster VAE decode

Unsloth shipped v0.1.900-beta on 2026-09-28 (PyPI unsloth 2026.9.12): Desktop runs Laya Decision models locally via a Jev-compatible /v1/systemone endpoint, adds a Skills CRUD editor and Library viewers for PDF/Office, and improves Apple Silicon with batched serving, structured outputs, and TurboQuant KV. Media pipelines claim ~4.5× LTX-2.3 clips, 1.7–6.3× VAE decode, and up to ~1 minute faster MiniMax-H3 first render with 25–29 GiB lower peak VRAM.

Unsloth v0.1.900-beta: local Laya Decision models + Skills editor; ~4.5× LTX-2.3 clips and up to 6.3× faster VAE decode

Key Takeaways

  • Shipped v0.1.900-beta — local Laya Decision API + Skills editor + Library viewers
  • LTX-2.3 ~4.5×; VAE decode 1.7–6.3×; MiniMax-H3 first render up to ~1 min faster, −25–29 GiB peak VRAM
  • Apple Silicon: batched serving, structured outputs, TurboQuant KV; ModelScope downloads
  • Docs: docs.unsloth.ai
Read full story
Linked Models & Agents:
⚡ Laya⚡ LTX-2.3⚡ MiniMax-H3⚡ Gemma-4🤖 Unsloth-Studio🌐 Unsloth
#Unsloth#Decision Model#Jev#Skills#Apple Silicon#MLX#LTX#Fine tuning
G
GitHub Next@GitHubNext·
👀139
🔥91
View on 𝕏 ↗

GitHub Next opens LocalJev: local Jev-compatible decision API on oMLX

GitHub Next open-sourced LocalJev (github.com/githubnext/localjev): a morning build of a “poor man’s Jev” for teammates without TypeSafe Jev access. It exposes a Jev-compatible POST /v1/systemone API in Bun/TypeScript, defaults to oMLX-backed local models such as DiffusionGemma, and reports evals across DiffusionGemma, Qwen MoE, Gemma 4 MoE, and Gemma 4 e4b/e2b (benchmark report in-repo). The LocalJev server works with normal Jev client wrappers, but probabilities are model-reported rather than logit reads—calibrate before consequential use.

Key Takeaways

  • Open-sources LocalJev: a local Jev-compatible POST /v1/systemone decision API (Bun/TypeScript).
  • Defaults to oMLX-backed local models such as DiffusionGemma, with multi-model evals in-repo.
  • Works with normal Jev client wrappers; probabilities are model-reported, not logit reads—calibrate first.
Read full story
Linked Models & Agents:
⚡ diffusiongemma⚡ qwen⚡ Gemma 4🌐 github
#LocalJev#GitHub Next#Jev#TypeSafe#oMLX#DiffusionGemma#local LLM#decision model
You've caught up with all stories