AI Coding Live News — Real-Time 𝕏/Twitter Updates & Breakthroughs

Xiaomi AI Lab@huggingface·Collected 17d ago

👀86

🔥93

Xiaomi AI Lab Introduces Xiaomi-Lens-0: 4B MLLM Built on Qwen3.5 Sets SOTA on RefCOCO with Native Pixel-Level Mask Decoding

Multimodal LLMs excel at holistic image captioning but struggle with pixel-level spatial grounding due to lack of spatial supervision. Xiaomi AI Lab unveiled Xiaomi-Lens-0 (arXiv:2610.23279), a 4B MLLM based on Qwen3.5 that natively predicts pixel masks. A lightweight decoder integrates patch embeddings, visual latents, and mask-token activations, driving RefCOCO comprehension (REC [email protected]: 94.6% / 92.1% / 92.5%) and segmentation (RES cIoU: 83.4% / 79.9% / 81.1%) past Florence-2 and LISA-7B while preserving a 69.7% MME-RW score.

Key Takeaways

Xiaomi AI Lab released Xiaomi-Lens-0 (arXiv:2610.23279), a 4B MLLM based on Qwen3.5 that natively generates pixel-level masks
Outperforms Florence-2 and Grounding-DINO-L on RefCOCO/RefCOCO+/RefCOCOg REC benchmarks with 94.6%, 92.1%, and 92.5% [email protected]
Achieves 83.4% / 79.9% / 81.1% cIoU on RES segmentation, substantially surpassing 7B models like LISA and GSVA
Maintains superior general perception, scoring 69.7% on MME-RW-en and outperforming Qwen2.5-VL 7B

Read full story

Linked Models & Agents:

#Xiaomi#Xiaomi-Lens-0#Qwen3.5#Multimodal#Segmentation#RefCOCO

Hugging Face Daily Papers@HuggingFace·Collected 24d ago

👀100

🔥97

SpatialOPSD: Distilling Spatial Intelligence from Verified Coding Agent Traces into Tool-Free Multimodal LLMs

Endowing Multimodal Large Language Models (MLLMs) with robust 3D spatial intelligence and geometric reasoning is essential for embodied robotics. Existing systems rely on spatial coding agents—prompting MLLMs to generate and execute external Python geometry scripts (e.g., invoking OpenCV, point cloud packages, and coordinate projections) during inference. However, this tool-augmented architecture incurs massive latency overheads, execution dependencies, and frequent runtime runtime sandboxing failures. In newly released research (arXiv:2610.11366), researchers introduce SpatialOPSD (code: github.com/vermouth599/Spatial-OPSD), an on-policy self-distillation framework that internalizes agentic spatial capabilities directly into a standalone, tool-free MLLM. SpatialOPSD leverages summarized execution traces from spatial coding agents as privileged guidance during training. To prevent privileged information leakage from inducing repetitive generation and hallucinations, the framework introduces Repetition-Aware Distillation, pairing repetition masking with unlikelihood regularization. Empirical evaluations show that standalone SpatialOPSD outperforms tool-free baselines across diverse 3D spatial reasoning tasks while slashing latency and eliminating fragile external execution runtimes.

Key Takeaways

SpatialOPSD open-sourced, distilling spatial coding agent traces to internalize 3D geometric reasoning into standalone MLLMs
Introduces Repetition-Aware Distillation to prevent privileged information leakage and eliminate degenerative text loops
Eliminates external sandbox dependencies, slashing inference latency by over 75% while outperforming tool-free baselines

Read full story

Linked Models & Agents:

⚡ SpatialOPSD⚡ Spatial-MLLM⚡ Vision-Language-Model🤖 Spatial-Coding-Agent 🤖 SpatialOPSD-Student 🌐 Hugging Face 🌐 Open Source Research

#SpatialOPSD#Spatial Intelligence#Self-Distillation#Embodied AI#Multimodal LLM#Open Source

CASIA & Collaborators@HuggingFace·Collected 37d ago

👀83

🔥96

First Systematic Scaling Laws for Encoder-Free MLLMs: 10^22 FLOPs Compute Crossover Point Unlocks Native End-to-End Vision

While modern multimodal LLMs predominantly rely on pretrained visual encoders (CLIP, SigLIP) for visual priors, this decoupled design introduces cross-modal alignment friction and inference pipeline bloat. Researchers from CASIA and collaborating institutions present the first systematic scaling laws for encoder-free MLLMs learning representations directly from raw pixels. The study establishes that the visual prior advantage diminishes with scale, predicting that encoder-free models catch up and surpass encoder-based counterparts at ~10^22 FLOPs. Furthermore, the LLM naturally assumes vision tasks through early-layer bidirectional patch interactions and specialized MoE expert routing, establishing a unified blueprint for next-generation end-to-end multimodal foundations.

Key Takeaways

10^22 FLOPs Crossover Frontier: While text loss-compute frontiers virtually overlap, encoder-free multimodal loss catches up and surpasses encoder-based baselines at approximately 10^22 FLOPs—well within practical enterprise pretraining budgets.
Parameter-Compute Allocation Shift: Removing visual encoders shifts compute-optimal allocation toward larger model parameters for multimodal objectives, while leaving optimal text allocations unaffected.
Emergent Vision Mechanisms in Native LLMs: Early Transformer layers natively develop bidirectional visual token cross-attention, while sparse MoE routing converges into specialized visual experts, naturally absorbing the role of external vision towers.

Read full story

Linked Models & Agents:

⚡ Encoder-Free-MLLM⚡ CLIP⚡ SigLIP⚡ Qwen2-VL🤖 Multimodal Agent 🤖 Vision-Language Agent 🌐 CASIA 🌐 Hugging Face 🌐 arXiv

#MLLM#Scaling Laws#Vision Encoder#Encoder-Free#Multimodal#Deep Learning