One of the early Google co-scientist papers proposes a really nice idea for a rather general problem: suppose we have a set of potential solutions for a given problem, and we want to know which solution is the most promising. In the article’s specific case, they have a set of potential scientific hypotheses and want to know which ones are most likely to be experimentally validated. Or we have briefings about some materials’ properties and want to know which material is likely to do best in our experiment. Or we have multiple potential job applicants, and we want to find the most promising one… See, it’s a very general idea.

Given this setup, can AI help us there? If the candidate set is small, a naïve solution is to simply prompt for a ranking. However, if you have many hypotheses and need to consider them carefully, prompting soon hits its limits, for instance due to context-length constraints or degraded reasoning when the model must compare too many candidates at once.

The proposed solution is pretty elegant. Conceptually, instead of comparing all candidates in one prompt, we play a tournament with our hypotheses. The hypotheses compete one-to-one. That is, instead of asking the AI to rank the full hypotheses list, we always give it exactly two hypotheses and ask it to find the more promising one. If every hypothesis is compared with every other hypothesis, we can order the list by the hypotheses’ win ratio to get the most promising ones to the top. Moreover, the process itself can be noisy or non-transitive, like A beats B, B beats C, yet C beats A, but this does not interfere with our overall ranking idea.

Now, if we have $N$ hypotheses and implement this naively, it becomes compute-intensive pretty quickly, as we have $\frac{N(N - 1)}{\mkern-2mu 2\mkern2mu}$ candidate pairings. However, there is a cleverer way to end up with a ranking than playing all pairings. The core idea is borrowed from chess and called a Swiss-system tournament. Instead of playing all potential matchings, at the start we take all hypotheses and split them into two groups, and members of these groups play against each other. Winners receive one point, losers no points. In the next round, winners play winners, and losers play losers. This is iterated multiple times so that groups of players with the same score face each other, avoiding repeated matchups. With this setup, the number of games played can now be controlled by the number of iterations. If we set that low, we can get a rough ranking quickly; if we set it higher, the ranking quality usually improves at the cost of playing more games. That way, we can now make the idea of the one-against-one ranking tractable by choosing a suitable number of iterations.

This is a sketch of the idea, and the co-scientist paper and the Wikipedia article on the Swiss-tournament system provide more insights. For instance, there are adaptations that account for player strength (e.g. Elo-style ratings), as well as subtleties in evaluating whether the approach actually produces good rankings.

What I love about it is that it is rather simple to explain as a concept and still powerful, as it allows us to establish a ranking for the many types of items that are comparable pairwise via prompting. Aside from this beauty, it also has the added benefit of being parallelizable, allowing us to greatly speed up the ranking. So far, I have only used it once for work with a hand-rolled implementation in Microsoft AutoGen, but I think this is going to be a more commonly used approach in multiple scenarios.