@agenticgirl covered aitofound/ScienceIDE: an eval for coding agents that compile but get the physics wrong. Agents edit pinned scientific codebases; a verifier recompiles, reruns cases, and checks numerical outputs and physical invariants. Inventory: 64 environments from 27 codebases, 2,812 tasks, 1,076 executable checks; hard set includes PLUTO, Athena++, MITgcm, and more. ~70 GitHub stars at fetch.

Key Takeaways

  • ✓Scores compile + scientific cases and physical invariants, not reference diffs.
  • ✓64 environments / 27 scientific codebases, 2,812 tasks, 1,076 executable checks.
  • ✓Open-source aitofound/ScienceIDE; ~70 GitHub stars at fetch.
ADSponsored