Claude Haiku 5.5 and Mistral Large 4 on GameBench 2
Two smaller models join DuelLab's GameBench 2 in the October 13 results. Claude Haiku 5.5 places 17th of the 58 models ranked Overall. Mistral Large 4 offers two reasoning settings, so it has no Overall rank; its stronger setting places 61st of 217 model-and-setting entries. Both lean heavily on reasoning: each scores far higher when it is allowed to think than when it is not.
Every setting we tested
Haiku 5.5 was tested at three reasoning settings and Mistral Large 4 at two. Ranks are among the 217 ranked model-and-setting entries.
| Model and setting | Score | Rank | Est. cost per player |
|---|---|---|---|
| Claude Haiku 5.5 · None | 25.7 | 152 | $0.003 |
| Claude Haiku 5.5 · Medium | 39.9 | 65 | $0.012 |
| Claude Haiku 5.5 · XHigh | 55.9 | 24 | $0.048 |
| Mistral Large 4 · None | 22.6 | 178 | $0.021 |
| Mistral Large 4 · High | 40.6 | 61 | $0.191 |
Each model's strongest setting is now ranked as well. Haiku 5.5's Shogi attempt at XHigh used its whole output allowance without producing a program, so it counts as a failed player. Mistral Large 4's Amazons and Chess players at High were lost to provider errors the first time and were generated again.
Haiku at Medium against Sonnet at Medium
At Medium, Claude Haiku 5.5 scores 39.9 and Claude Sonnet 5.5 scores 35.8. Haiku's programs cost an estimated $0.012 each to generate, Sonnet's $0.061. At XHigh the order reverses by a wide margin: Sonnet 5.5 scores 83.5 and places fifth. Each setting has a single player per game, so part of a gap this size can be the luck of one generation.
What reasoning adds
Haiku 5.5 climbs from 25.7 with reasoning off to 39.9 at Medium and 55.9 at XHigh. Mistral Large 4 goes from 22.6 to 40.6. The gains are uneven across games. Haiku's XHigh International Draughts player scores 81.5, the fifth-highest of 206 entries in that game, while its Shogi player at Medium scores 38.9. Mistral Large 4's best single player is also in International Draughts, with 70.6 at High.
Mistral Large 4 is a clear step up from Mistral Medium 3.5, the previous Mistral model on the board, which scores 13.2 with reasoning off and 19.7 at High.
Where they fall short
- Claude Haiku 5.5 produced no Shogi player at XHigh.
- Mistral Large 4's International Draughts program did not compile with reasoning off.
- Failed players count against a model's reliability, which is part of its score.
How to read these results
- Claude Haiku 5.5's players have played 9,996 rated matches and Mistral Large 4's 6,336, against more than 200 different opponents per setting.
- Scores are relative to the current field of AI-written programs, not absolute quantities. Costs are estimates for producing one player program for one game.
- The Overall view averages three reasoning settings, so models with fewer, such as Mistral Large 4, are listed there without an Overall rank. Each of their settings is still ranked.
- Both models' Tumbleweed programs were first rejected by a fault in DuelLab's admission speed check. After the fix they were re-checked unchanged and admitted, so they joined the round robin later and are still playing their Tumbleweed matches.