Artificial Analysis Cyber Index: Tencent Hy4 Scores 58 to Rank Third and Tops CWE-Bench-AA Vulnerability Fixing at 76%, at $0.92 per Task

Background and the problem

Coding agents are increasingly used for code audits, but general leaderboards say little about whether a model can actually find and fix a vulnerability. Artificial Analysis (AA) built the Cyber Index, separate from its Intelligence Index, to measure defense only: finding vulnerabilities, reproducing and validating them, and patching without breaking existing behavior. Models are never asked to write working exploits. Tencent shipped Hy4 on November 4, AA already reported its Intelligence Index score, and this release adds the cyber dimension.

Architecture and how it works

The Cyber Index is an equally weighted average of three evaluations: CWE-Bench-AA (based on Collinear AI's CWE-bench), DeepsecBench-AA (based on Vercel's DeepsecBench) and CyberGym-E2E-AA (based on CyberGym-E2E from Berkeley RDI). CWE-Bench-AA has 120 private tasks covering all ten OWASP Top 10 (2025) categories across C/C++, Go, Java, JS/TS, Python and Rust. Each task names an area of concern but not the exact location. Models work in an offline sandbox on AA's open-source Stirrup agent harness with a 2-hour limit per task, and a task only passes when a programmatic check confirms the exploit no longer works and legitimate behavior still does. There is no partial credit.

Benchmarks and measured results

All figures are AA's independent results. Hy4 scores 58 on the Cyber Index and ranks third, behind the trusted-access-only GPT-6 Sol (Daybreak Blue, max) at 69 and Kimi K3.1 Pro at 64, and ahead of DeepSeek V4.1 Flash 1017 (max) at 57. On CWE-Bench-AA Hy4 ranks first at 76%, ahead of DeepSeek at 69% and Grok 4.7 (xhigh) at 68%. It averages $0.92 and about 64k output tokens per task. AA's scores are not directly comparable to Collinear's own leaderboard at cwe-bench.com, which runs each vendor's own agent product, four trials per task, with a partial-credit track.

Getting started for developers

Hy4 is available on Tencent Cloud and OpenRouter (model tencent/hy4, 1M context). To try it for dependency or codebase security scans, call it through OpenRouter from tools that accept custom models, such as Claude Code or Cline. Tencent says the final Hy4 weights will be open-sourced; what is downloadable on Hugging Face today is Hy4-preview under Apache 2.0. Benchmarks describe capability only, so patches in production still need human review and regression tests.