CellType, a YC W26 startup, released BioSimBench 1.0 on November 4. It uses real lab experiments to test whether language models can predict which genes go up or down after a cell is hit by a drug, a gene knockdown or an immune signal. Among 12 frontier models, Gemini 4 Argon ranks first with an overall score of 0.184, ahead of Claude Sonnet 5.5 (0.160) and GPT-6 Astra (0.159), while two lab replicates agree at 0.462. A 27B model CellType trained in its BioSimGym RL environments beats the best frontier model on all four prediction tracks.
Key Takeaways
- ✓BioSimBench 1.0 has five tracks: new drugs and cell contexts, gene knockdown, drug response, immune signals, and a tool-using research agent; test answers are kept private.
- ✓The 12 frontier models score between 0.110 and 0.184 overall, with Gemini 4 Argon first and DeepSeek V4 Pro last; two lab replicates agree at 0.462, so the gap is still large.
- ✓CellType 27B scores 0.366 on new drugs and cell contexts and 0.203 on drug response, about double the best frontier score of 0.103.
- ✓Data processing uses NVIDIA RAPIDS-singlecell, at least 4x faster for the whole pipeline; models were trained with NeMo RL and BioSimGym follows NeMo RL environment interfaces.
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and industry pain points
Drugs, genes and cell contexts combine into far more experiments than any lab can run. Language models write fluently about biology, but there has been no public test scored against real lab data on whether they can say which genes go up or down after a specific cell is perturbed. CellType, founded by the Yale team behind Cell2Sentence, released BioSimBench 1.0 on November 4 to measure exactly that.
Architecture highlights and mechanisms
A model gets a cell's baseline expression plus an intervention (a drug, a CRISPRi knockdown or an immune signal), lists upregulated and downregulated genes, and is scored against lab measurements with a perturbation-specific F1 that removes genes moving under any perturbation. The five tracks hold out new drugs, cell lines, knocked-down genes or donors, and test answers stay on CellType's scoring servers. BioSimGym is the matching set of RL training environments.
Benchmarks and measured comparisons
On the official leaderboard Gemini 4 Argon scores 0.184 overall, Claude Sonnet 5.5 0.160 and GPT-6 Astra 0.159; Gemini 3.1 Pro, GPT-6.1 Sol and Gemini 3.8 Flash tie at 0.153, Claude Opus 5.5 scores 0.145 and DeepSeek V4 Pro 0.110. Two lab replicates agree at 0.462. CellType 27B (a vote of 9 answers) scores 0.366, 0.216, 0.203 and 0.267 on the four prediction tracks, beating the best frontier model on each; these are CellType's own evaluations.
Developer adoption and getting started
BioSimBench is public, and researchers can submit a model for evaluation or apply to train in BioSimGym from the official site. AI-for-science teams can use it to check whether a model's biological reasoning holds up against measured outcomes; the BioSimGym environments themselves are a commercial product CellType licenses to drug discovery teams.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.