Claude Sonnet 5.5 jumps from 70th to 5th with more reasoning
Most models in GameBench 2 write their game-playing programs at three reasoning settings, and each setting is ranked on its own. No model in the October 6 results depends on the setting as much as Claude Sonnet 5.5. At XHigh it is fifth of 191 ranked entries. At Medium it is 70th, and at Low 125th.
| Setting | Score | Rank | Est. cost per player |
|---|---|---|---|
| Low | 28.1 | 125 | $0.04 |
| Medium | 36.4 | 70 | $0.06 |
| XHigh | 83.6 | 5 | $0.55 |
Where the jump comes from
The XHigh programs are stronger in every game, and the gap is largest in Chess and Shogi. In Chess, the Medium player won 5 of its 282 rated matches, drew 214 and lost 63, for a score of 7.4. The XHigh player won 259, drew 18 and lost only 3, for 96.6. Shogi goes from 15.9 to 88.6, and Hex from 26.9 to 90.4. The XHigh Tumbleweed player never lost a match, with 292 wins and 2 draws.
| Game | Low | Medium | XHigh |
|---|---|---|---|
| Go | 38.5 | 38.5 | 64.4 |
| Chess | 12.9 | 7.4 | 96.6 |
| Shogi | 28.6 | 15.9 | 88.6 |
| Xiangqi | 15.0 | 9.1 | 62.3 |
| Hex | 21.0 | 26.9 | 90.4 |
| International Draughts | 28.8 | 70.4 | 81.2 |
| Game of the Amazons | 44.6 | 61.7 | 85.2 |
| Tumbleweed | 35.6 | 61.3 | 100.0 |
The extra reasoning costs money. An XHigh player costs about nine times as much to generate as a Medium one, $0.55 against $0.06. Claude Opus 5.5 at Medium scores 82.9, close to Sonnet 5.5 at XHigh, for $0.33.
Most models play best at their highest setting
Forty-four models are ranked at all three of their tested settings. For 33 of them the highest setting scores best. The middle setting is best for eight and the lowest for three. In July, with fewer models, 12 of 15 scored best at the highest setting (see In GameBench 2, more reasoning is not always better).
Where more reasoning loses, the losses are small. Grok 4.6 scores 52.9 at Medium and 48.5 at XHigh. Nemotron 3 Ultra, DeepSeek V4 Pro and Claude Opus 4.5 each lose between 2.8 and 4.6 points at their highest setting compared with the middle one. The gains are much larger. After Sonnet 5.5's 47.2 points, the next biggest jumps from the middle to the highest setting are about 20 points.
One program per game
Each setting produces one player per game, so a single result can partly reflect the luck of one generation. A model that writes one weak Chess program would score lower in Chess even if its typical program were stronger. DuelLab does not yet estimate this variation, so the figures describe the programs each model wrote. They are not a fixed property of the model.