Online and Stochastic Optimization beyond Lipschitz Continuity: A Riemannian Approach

Kimon Antonakopoulos (1) , Elena Veronica Belmega (2) , Panayotis Mertikopoulos (1)

1 POLARIS - Performance analysis and optimization of LARge Infrastructures and Systems
2 ETIS - UMR 8051 - Equipes Traitement de l'Information et Systèmes

Motivated by applications to machine learning and imaging science, we study a class of online and stochastic optimization problems with loss functions that are not Lipschitz continuous; in particular, the loss functions encountered by the optimizer could exhibit gradient singularities or be singular themselves. Drawing on tools and techniques from Riemannian geometry, we examine a Riemann-Lipschitz (RL) continuity condition which is tailored to the singularity landscape of the problem's loss functions. In this way, we are able to tackle cases beyond the Lipschitz framework provided by a global norm, and we derive optimal regret bounds and last iterate convergence results through the use of regularized learning methods (such as online mirror descent). These results are subsequently validated in a class of stochastic Poisson inverse problems that arise in imaging science.

Domaines: Optimisation et contrôle [math.OC]

Kimon Antonakopoulos, Elena Veronica Belmega, Panayotis Mertikopoulos. Online and Stochastic Optimization beyond Lipschitz Continuity: A Riemannian Approach. ICLR 2020 - International Conference on Learning Representations, Apr 2020, Addis Ababa, Ethiopia. pp.1-20. ⟨hal-03043671⟩