Comet Opik 2.2.84 (2026-09-29) lets online LLM-as-a-Judge rules call TypeSafe Jev end-to-end via OpenRouter POST /api/alpha/decisions (not chat/completions). Each Boolean metric becomes a yes/no question answered in one call; score 1 if probability ≥0.5, with Probability: 0.93 in the reason. Backend adds OpenRouterDecisionsClient (workspace OpenRouter key, 429/5xx retries) for ~typesafe/jev-latest / typesafe/jev-1.13. Also: Annotation Queue automation + item provenance. ~8 commits / 104 files vs 2.2.83.

Key Takeaways

  • ✓Release 2.2.84; ~8 commits / 104 files vs 2.2.83
  • ✓PR #8540: online judges call Jev via OpenRouter Decisions API
  • ✓Boolean metrics → yes/no questions; score 1 if p≥0.5; reason shows Probability
  • ✓Models: ~typesafe/jev-latest, typesafe/jev-1.13
  • ✓Queue automation + item provenance; trace stats / thread_id fixes
Comet Opik 2.2.84: online eval rules call Jev via OpenRouter Decisions API; queue automation + provenance
🖼️Official Media
Click to view high-res
⚡

Key Decision Metrics at a Glance

TL;DR Verdict
🏆 Benchmark RankRank
See how it ranks on the leaderboard
Full Leaderboard→
💰 Cost & PlansCompare
API & Subscription
Compare Plans↗
🛠️ Ecosystem & IDERules
opik · jev · openrouter
Setup IDE Rules↗
🧭

Turn your technical choice into a development budget

Compare 40 dev plans & simulate token costs vs $20/mo subscriptions

⚔️ Compare Model→💰 Compare plans & limits↗
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🐙
Opik 2.2.84 release
github.com/comet-ml/opik/releases/tag/2.2.84
↗
🐙
PR #8540 Jev via OpenRouter Decisions
github.com/comet-ml/opik/pull/8540
↗
🐙
Compare 2.2.83...2.2.84
github.com/comet-ml/opik/compare/2.2.83...2.2.84
↗
📖
Opik documentation
www.comet.com/docs/opik/
↗
🚀
TypeSafe (Jev)
typesafe.ai/
↗
🐙
comet-ml/opik
github.com/comet-ml/opik
↗
🔬

In-Depth Technical Analysis

Background

Online LLM-as-a-Judge rules often force Boolean checks through chat completions. TypeSafe Jev is a decisions model: OpenRouter serves it on POST /api/alpha/decisions and rejects chat/completions—Opik needed a dedicated scoring path.

What shipped

2.2.84 (PR #8540) routes Boolean online rules through OpenRouterDecisionsClient for ~typesafe/jev-latest / typesafe/jev-1.13. One Decisions call answers all yes/no questions; score 1 if p≥0.5. Also: queue automation + provenance; trace stats / thread_id fixes.

Benchmarks

No public judge-accuracy table. Signal: ~8 commits / 104 files vs 2.2.83. Calibrate the 0.5 threshold on your labels.

Get started

Upgrade to 2.2.84, set the workspace OpenRouter key, pick a Jev model in online eval rules. Docs: Opik, repo, TypeSafe.

Action HubReady to adopt this in production?

Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.

⚔️ Launch VS Compare→💳 View Plans↗
Up NextOpenRouter ships Model Router Benchmarks: 7 routers vs 6 suites on quality, speed, and cost
Read Now→