Why GameBench 2 now uses a Glicko-style rating
Until October, GameBench 2 rated player programs with sequential Elo, the system that updates two ratings after every match. Starting with the October 6 release, ratings come from a Glicko-style fit over all of a game's matches at once. The main reason is that under sequential Elo, the order in which matches happened to be played changed the results.
The problem with sequential Elo
Sequential Elo moves two ratings a little after each match, so the same set of results can end in different ratings depending on their order. That suits a chess club, where recent games should count more. In GameBench 2 it is just noise, because a generated program never changes after it is written. Its fifth match tells us no more about its strength than its first.
To measure the effect, we replayed the September 25 evidence in five different orders. The 204,246 rated results stayed the same each time, and only their order changed. A typical player program's score moved with a standard deviation of about 2.5 points. About a quarter of programs moved by more than 5 points. Model scores were steadier, at about 0.7 points, but 41 to 51 of the 73 model ranks changed in each reshuffle.
Sequential Elo also lags behind strong newcomers. Every program starts at 1500 and can move at most 32 points per game, so a strong new model needs many matches before its rating catches up. In held-out tests, programs from a newly added model were still rated too low after 48 games. The Glicko-style fit cut that error by roughly two thirds or more.
How we chose the replacement
We compared two order-independent candidates, a Glicko-style fit and a Bradley–Terry model. Before running the comparison, we wrote down the rule that would decide between them. If neither passed, Elo would stay. The deciding run on October 4 used a validation design fixed in advance.
Both candidates passed, and their results were effectively tied. The widths of their interval estimates differed by less than 0.002 on a 0-to-1 scale. The pre-registered rule then chose the Glicko-style fit. On held-out results it also predicted match outcomes better than sequential Elo in all eight games, with a log loss of 0.4906 against 0.5026. A fully converged Bradley–Terry fit predicted equally well or slightly better, so we call the choice a near-tie, not a clear win.
What the new rating does
- Every player program starts at 1500 with a rating deviation of 350. All rated matches in a game are refitted together in 12 passes, so their order does not matter.
- A win counts 1, a draw 0.5 and a loss 0. Each pass re-centres the field on 1500, and the deviation is kept between 30 and 350.
- The score uses a conservative rating, the rating minus twice the deviation. Each game's conservative ratings are scaled from 0 to 100 within the current field. The rest of the scoring is unchanged. A model's score still averages its playable games and is reduced when some of its programs fail.
- It is a Glicko-2-style kernel rather than the full Glicko-2 system, with no volatility parameter and no rating periods.
The full description is on the methodology page.
What changes for readers
- Scores and ranks from October 6 onwards are not directly comparable with earlier releases, which used sequential Elo. The site shows no rank-change arrows across the change. On the same October 4 evidence, switching the rating alone moved a typical model-and-setting entry by three places and 1.6 points, and the same ten entries stayed in the top ten.
- Earlier releases are unchanged. Each remains available, and citable, as it was published.
-
Data fields named
raw_elokeep their name for compatibility. From October 6 they hold the Glicko-style rating, as the data dictionary explains.