2026-11-03 · Economics · All posts

What a strong game-playing program costs

In GameBench 2, an AI model writes one program for each of eight strategy games, and the programs then play their matches without calling the model again. Generating the program, with any repairs, is the only model cost. In the 3 November release, gb2-20261103-09b2218cbd85, the ten highest-ranked model-and-setting rows cost between $0.030 and $5.15 per program, a 174-fold range. Claude Opus 5.5 at XHigh ranks first with 94.9 for $1.99 a program. GPT-6.1 Sol at XHigh scores 91.9 (4th) for $0.267, and Pareto 26.10 scores 89.7 (7th) for $0.030.

The best-value line

Of the 242 ranked rows, 170 have complete cost records. Among those, 9 form a best-value line (for each of them, no other such row scores as high for the same cost or less), the same line the model pages on the benchmark site draw. Its upper end has five rows: GPT-6.1 Terra at None (79.4 for $0.027), Pareto 26.10, Sol at XHigh, GPT-6 Astra at XHigh (94.0 for $1.30) and Opus at XHigh.

From Terra at None to Pareto 26.10 the line climbs 10.3 points at almost the same cost. The three steps after that add 2.3, 2.1 and 0.8 points, for 9.0, 4.9 and 1.5 times the cost of the row before. Between Pareto 26.10 and Opus at XHigh, the score rises by 5.2 points and the cost 67-fold.

Among the ranked rows with complete costs, 6 score 90 or more, and the cheapest of them is Sol at XHigh. Of the 12 that score 85 or more the cheapest is Pareto 26.10, and of the 34 that score 70 or more it is Terra at None. Pareto 26.10 has one tested setting, because its provider offers no reasoning controls, and it has no Overall rank.

Points near the top

The 0–100 score is a program's expected share of the points against a fixed set of reference programs, so it compresses near 100. Two programs that both beat the reference programs almost every time both score close to 100, even when one of them clearly beats the other; our technical report shows this game by game. For setting scores, which average eight games, a step from 89.7 to 94.9 stands, roughly, for a larger difference in playing strength than a step of the same size in the middle of the scale.

The most expensive ranked row with a complete cost is Claude Fable 5.1 at XHigh, second with 94.1 for $5.15 a program. Opus at XHigh scores 0.7 points more for 39% of that. Most rows rest on one generation attempt per game (206 of the 242 ranked rows), and the release publishes no uncertainty interval for a whole row. We would not rely on a gap of a point or two between neighbouring rows as an order.

How the costs are measured

A row's cost per program is the mean cost of generating one program, over its games, with repairs and failed attempts included. Each combination of model, game and setting counts once; repeated runs of the same combination are averaged first. We use a reconciled bill where we have one, then the cost the provider reports, then an estimate from versioned public list prices. Because the basis differs between rows, cost comparisons are approximate. Among all ranked rows, 154 have provider-reported costs, 87 have list-price estimates and 1 mixes the two. Every row named above except Pareto 26.10 is a list-price estimate, and so are 8 of the top ten.

For 72 rows, some attempts have no cost record, and the published cost is a known subtotal. We leave them off the line, as the model pages do, and the chart draws them hollow. Cost plays no part in the ranking.

Each model page on the benchmark site has a "Reasoning settings compared" section that places the model's settings on this plot and names, for each setting, any ranked setting that scores higher for the same cost or less.