Can we build an AI benchmark that never saturates?

What happens when AI models become good enough to pass our tests, and the results stop meaningfully distinguishing them?

For DuelLab, our bet is that deeply strategic games offer a different kind of challenge. Understanding the rules and writing a working player are only the beginning. Beyond that lies the problem of discovering better strategies and turning those strategies into better game-playing programs.

A benchmark becomes saturated when its results stop usefully distinguishing the models we want to compare. Our ambition is to build an evaluation that remains useful even as AI advances far beyond today's capabilities. The central idea is not that we must continually replace the test. It is that the same deep games can keep rewarding better solutions.

A working player is not the end of the problem

DuelLab asks models to write game-playing programs. We provide the rules and an interface, check the resulting programs, and run them against one another. The programs make the moves during matches—not the language models that wrote them.

This changes what success means. Producing valid code is necessary, but it is not the finish line. A program must make decisions that survive contact with an opponent.

Consider two hypothetical players that both pass the code checks and make legal moves. One follows a few sensible heuristics. The other recognizes when those heuristics are misleading, anticipates the opponent’s response, and finds a plan that pays off several moves later. A check that merely asks whether the programs work could accept both. Competition can expose the difference.

Our interest is in how far that distinction can extend. Beyond a competent player, there may be a stronger evaluation of positions, a more effective search method, or a strategic idea the earlier program never considered. Being better than today’s opposition need not mean that those possibilities have been exhausted.

During matches, the programs compete under the same execution conditions for each game. Within those limits, improvement can come from searching more selectively, evaluating positions more effectively, or avoiding wasted work. The challenge is not simply to calculate more, but to write a program that uses its calculation better.

Stronger opponents reveal the difference

Strategic depth creates opportunities for improvement. Competition gives us a way to test whether a proposed improvement actually helps.

A weakness may go unnoticed against an opponent that cannot exploit it. A stronger opponent can make that same weakness decisive. A strategy that looked convincing may need to be refined when it meets better resistance.

A useful long-term comparison needs both established opponents and new challengers. Earlier players provide reference points; new players can expose weaknesses that the old field missed. The aim is not simply stronger opposition, but opposition that exposes different weaknesses.

DuelLab’s scores are relative to the evaluated field: a score of 100 on a game does not mean perfect play. Scores can change when opponents or evidence change, even if a program is unchanged. What matters is whether the matches continue to reveal meaningful differences in playing strength—not simply whether the displayed numbers change. (Scoring methodology)

We do not need to invent the best strategy ourselves

DuelLab’s deterministic engine applies the rules and records match outcomes. A human or another AI does not have to judge which program seemed more intelligent.

That matters for our long-term ambition. A model could write a player whose strategy we would never have discovered ourselves, yet its performance could still be tested through competition. We do not need to know the best strategy in advance to determine the outcome of a match.

This does not make interpreting the evidence effortless. We still need appropriate opponents, comparable conditions and sufficient coverage. But our own ability to invent strategies need not set the ceiling on the strategies we can evaluate.

Unfamiliar games broaden the challenge

Strategic depth is our main answer to saturation. Unfamiliar games address a related question: how well can a model adapt its knowledge to less familiar rules?

Tumbleweed is an interesting part of that investigation. Created in 2020, it is a territory game built around lines of sight and stacks. It already has published strategic material, including opening theory and local tactics, so we should not treat it as a blank slate. Nor can leaderboard results tell us how much of that material a model encountered during training.

Tumbleweed is already exposing substantial differences between some of our strongest tested models. In the September 10 standings, Claude Fable 5.1 and GPT-6 Astra, both at XHigh, are closely matched on the chess leaderboard, scoring 100.0 and 98.9. On Tumbleweed, their scores are 100.0 and 68.5, with Astra placing 18th. A model near the top of the overall benchmark does not automatically produce a leading player for every game. This suggests that lesser-known games can expose weaknesses that strong results elsewhere would leave hidden—and provide a meaningful challenge even for today’s leading models.

We are also working on entirely new games. The aim is not novelty alone, but rules that create difficult decisions and reward better strategy. A renamed familiar game would not achieve that. We still need to demonstrate the strategic depth and evaluation value of the games we create.

Transferring a useful algorithm to an unfamiliar game is a success, not a loophole.

What the results show

In the September 10, 2026 standings, GPT-6 Astra and Claude Fable 5.1 occupy the top four officially ranked reasoning variants across the eight-game suite. Their Medium variants rank above every officially ranked entry from the other models. (Published standings; dated release)

Astra offers another illustration. Across its three tested reasoning settings and eight games, all 24 recorded runs produced working programs on the first attempt. Yet its aggregate competitive scores differed: 75.3 for Low, 79.4 for Medium, and 91.0 for XHigh. A check that only asked whether these programs worked would give the three settings the same result. Competition did not. (Astra results)

Those results show that the evaluation distinguishes the tested systems today. They do not tell us where the eventual ceiling lies.

Explore the current standings, read the methodology, or inspect the September 10 release data. We welcome concrete strategies, game ideas and evaluation failures that would put this approach to the test.