What changed in GameBench 2 scoring v17
From the October 16 release, GameBench 2 uses scoring version 17. It changes how ratings are fitted, what the 0–100 score measures, and how failed programs count. It replaces the Glicko-style rating introduced on October 6.
What changed
- Ratings. Each game's ratings come from B1, a Bradley–Terry model fitted over all of the game's matches until it converges. Nothing is deducted for rating uncertainty.
- The 0–100 score. A game score is 100 × the points a program is expected to win (a win counts 1, a draw ½) against a fixed reference field: the programs of every model, each model with equal weight. The field stays fixed until we announce a new version; strengths are refitted at every release.
- Failed programs. Each attempt whose model-written code failed counts as 0 in its game, beside the working programs. The separate success-rate factor, which lowered a model's whole score for its share of failures, is gone.
- Faults during a match. A crash or an illegal move pinned on one program counts as that program's loss.
- Our own failures. Attempts that failed because of our harness, such as local errors or replies cut short by an output limit we set below the model's maximum, no longer count against the model. Cut-short replies were re-run at the model's full limit where the original request could be repeated; otherwise the game is left out.
- Ranking. A model-and-setting row is ranked only when all eight games have an evaluated attempt: a program, or a failure in the model's own code.
Why
The old scale stretched each game between its weakest and strongest program, so one program joining or leaving could move everyone else; the fixed field changes only when we announce it. B1 won a comparison set up in advance on October 14: on held-out pairings it predicted results no worse than the Glicko-style kernel, and it converges instead of stopping after a fixed number of passes. Counting failures game by game treats them the same way everywhere.
How to read the new numbers
- The typical score is about 14 points higher than in the previous release. Most of that comes from the new scale: the old one put each game's weakest program at 0, while on the new one even weak programs take some points. It is not a change in quality.
- Within each game, the score keeps programs in exactly the order of their ratings; we checked all eight games.
- The top of Overall is a near-tie: GPT-6 Astra 89.4, Claude Opus 5.5 88.6, Claude Fable 5.1 88.4. In the previous release Claude Opus 5.5 led GPT-6 Astra 78.1 to 77.9. The two swapped at the change of scale, not because of new matches: once a program wins almost every game, a larger rating lead adds little to its expected points. Read the top three as level.
- Nine model-and-setting rows ranked in the previous release are now unranked because a game has no evaluated attempt; in most of them, our harness cut a reply short and the request can no longer be repeated. Their scores average their other games. Two more models are now ranked Partial 3/3 in Overall because at least one of their settings is unranked.
The methodology page has the full description.