PAC-Bayes unleashed: generalisation bounds with unbounded losses

Maxime Haddouche (1, 2) , Benjamin Guedj (3, 4, 5, 6, 2) , Omar Rivasplata (4, 3, 2) , John Shawe-Taylor (4, 3, 2)

1 ENS Paris Saclay - Ecole Normale Supérieure Paris-Saclay
2 Inria-London - The Inria London Programme
3 UCL - University College of London [London]
4 UCL-CS - Department of Computer science [University College of London]
5 Inria-CWI - Inria-CWI
6 MODAL - MOdel for Data Analysis and Learning

We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems where the loss function is bounded (typically assumed to take values in the interval [0;1]). In order to relax this assumption, we propose a new notion called the \emph{special boundedness condition}, which effectively allows the range of the loss to depend on each predictor. Based on this new notion we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.

Domaines: Machine Learning [stat.ML]; Apprentissage [cs.LG]; Théorie [stat.TH]

Maxime Haddouche, Benjamin Guedj, Omar Rivasplata, John Shawe-Taylor. PAC-Bayes unleashed: generalisation bounds with unbounded losses. Entropy, 2021, ⟨10.3390/e23101330⟩. ⟨hal-02872173⟩