Judging competitions and benchmarks: a candidate election approach

Adrien Pavao (1) , Michael Vaccaro (2) , Isabelle Guyon (1)

1 TAU - TAckling the Underspecified
2 LISN - Laboratoire Interdisciplinaire des Sciences du Numérique

Machine learning progress relies on algorithm benchmarks. We study the problem of declaring a winner, or ranking "candidate" algorithms, based on results obtained by "judges" (scores on various tasks). Inspired by social science and game theory on fair elections, we compare various ranking functions, ranging from simple score averaging to Condorcet methods. We devise novel empirical criteria to assess the quality of ranking functions, including the generalization to new tasks and the stability under judge or candidate perturbation. We conduct an empirical comparison on the results of 5 competitions and benchmarks (one artificially generated). While prior theoretical analyses indicate that no single ranking function satisfies all desired properties, our empirical study reveals that the classical "average rank" method fares well. However, some pairwise comparison methods can get better empirical results.

Domaines: Intelligence artificielle [cs.AI]; Statistiques [math.ST]; Machine Learning [stat.ML]

Adrien Pavao, Michael Vaccaro, Isabelle Guyon. Judging competitions and benchmarks: a candidate election approach. ESANN 2021 - 29th European Symposium on Artificial Neural Networks, Oct 2021, Bruges, Belgium. ⟨hal-03367857v2⟩