2026-10-22 · Benchmark finding · All posts

Writing code that wins

Writing a program that runs is one achievement, and writing one that plays well is another. GameBench 2 asks AI models to turn the rules of eight classic strategy games into player programs, then makes those programs play each other. The language models never choose a move during a game; their programs do.

Our technical report on the 20 October 2026 release covers 85 models, the 1,855 programs they wrote and 428,924 rated matches between those programs. The full report has the methods and every caveat.

What a model is asked to do

For each game, a model gets one prompt: the rules, a small programming interface, a minimal code skeleton and a few notes on strategy. It answers in a single response with one Rust source file, with no tools, code execution or web access. The host tells the program which moves are legal, so the model does not have to implement the rules, but there are no search or evaluation helpers, and any look-ahead the program does it has to build itself. If the code does not compile or fails our checks, the model gets at most two repairs with the error messages. It never sees how its program plays.

Every admitted program then plays every other program of its game, once from each side. Ratings come from those results, and each rating becomes a score from 0 to 100: the program's expected share of points against a fixed set of reference programs. A model's Overall score is the average of its scores at a low, a middle and a high reasoning setting of its own, typically no reasoning or Low, then Medium and XHigh. Where a model offers both XHigh and Max, we use XHigh.

A close race at the top

Three models are nearly level. GPT-6 Astra leads with 89.6, ahead of Claude Opus 5.5 at 88.6 and Claude Fable 5.1 at 88.5. GPT-6.1 Sol follows at 85.8, and then the field drops away: Claude Opus 5, in fifth place, scores 78.1. We have not yet validated uncertainty intervals for individual models, so we treat the order of the top three as unresolved.

Reasoning effort and cost

From its lowest reasoning setting to its highest, the median model gains 15.3 points, and 52 of the 59 ranked models improve. The share of attempts that fail on the model's own code barely moves (9.9% at the lowest setting, 9.7% at the highest), while the strength of the working programs rises by a median 14.4 points, so the gain comes mainly from stronger programs rather than fewer failures.

For the 56 models whose highest setting costs more than their lowest, the step up costs a median $0.19 more to generate the program for one game (costs are provider-reported or estimated from list prices). Measured as score per dollar of generation cost, the lowest setting is the best value for 46 of 57 models, while the highest setting gives the strongest programs.

Leaders by game

No model leads everywhere. The strongest programs in the eight games come from 6 different models, and the best Go program, from GPT-5.4 with a score of 97.8, came from its middle setting. Several of these leads are close.

Progress by release date

We also placed each model at its release date. Among the models we evaluated, the best one released by October 2025 scores 42.9 in this release, and the best of all scores 89.6, a rise of 46.7 points. Models released in the last six months account for 34.2 of them. This is a cross-section of a single release rather than a history of past leaderboards, and only 3 of the dated models came out by October 2025, so the size of the rise depends on which older models we chose to evaluate.

Every new best came from two developers. Of the 9 models that beat every model released before them, 5 are from OpenAI and 4 from Anthropic.

What the scores mean

The 0 to 100 score is a share of points against a fixed field, so it bunches up near the top. In 5 of the 8 games the best program scores within 5 points of the fifth-best, yet its rating implies that it would take between 73.5% and 90.0% of the points in direct matches against it. At their highest settings the strongest models already score up to 95.2. The ratings underneath have no fixed ceiling while games stay decisive, and the games leave room, because no perfect strategy is known for any of the eight at the board sizes we use and most of them are proven to be computationally hard in their generalized forms. Fixed reference players and new games are how we plan to keep the 0 to 100 scale informative.

Does it tell us anything beyond games?

GameBench ranks models broadly like the popular leaderboards. Across eight comparisons, rank correlations run from 0.48 to 0.86, highest with LMArena's Agent Arena and the Epoch Capabilities Index (0.84).

Before seeing any results, we also wrote down a stricter test. With each model's whole developer held out, does adding GameBench to LiveBench's own scores improve predictions of LiveBench's agentic coding results? On that target neither GameBench nor anything else we tried beat a constant guess, so the test could not tell either way. In one of six secondary comparisons, on LiveBench 2.0's coding tasks built from real pull requests, adding GameBench cut the prediction error by 24%.

What we don't know yet

Most combinations of model, game and setting have one generation attempt, so a setting's score usually rests on eight programs, one per game, and another run could come out differently. Even with samples this small, GameBench ranks models much like well-established benchmarks.

The full report has the methods, the tables and the limits in detail, and the current rankings are on the GameBench leaderboard.