Empirical Risk Minimization with Generalized Relative Entropy Regularization

Minimisation du Risque Empirique avec Régularisation par l’Entropie Relative Généralisée

Samir M. Perlaza (1, 2, 3) , Gaetan Bisson (2) , Iñaki Esnaola (4, 3) , Alain Jean-Marie (1) , Stefano Rini (5)

1 NEO - Network Engineering and Operations
2 GAATI - Laboratoire de Géométrie Algébrique et Applications à la Théorie de l'Information
3 ECE - Department of Electrical and Computer Engineering [Princeton]
4 ACSE - Department of Automatic Control and Systems Engineering [ Sheffield]
5 Department of Electrical and Computer Engineering [Taiwan]

The empirical risk minimization problem with relative entropy regularization (ERM-RER) is investigated considering that the reference measure is a σ-finite measure instead of a probability measure. This generalization allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. In this setting, the interplay between the regularization parameter, the reference measure, the risk function, and the expected empirical risk induced by the solution of the ERM-RER problem, which is proved to be unique, is characterized. For a fixed dataset, the empirical risk is shown to be a sub-Gaussian random variable, when the models follow the probability measure that is the solution to the ERM-RER problem. The sensitivity of the expected empirical risk to deviations from the solution of the ERM-RER problem is studied and upper and lower bounds on the expected empirical risk are provided. Finally, it is shown that the expectation of the sensitivity is upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets.

Domaines: Machine Learning [stat.ML]; Théorie de l'information et codage [math.IT]; Intelligence artificielle [cs.AI]; Théorie de l'information [cs.IT]; Apprentissage [cs.LG]

Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Generalized Relative Entropy Regularization. [Research Report] RR-9454, Inria. 2022. ⟨hal-03560072v2⟩