Integration Methods and Accelerated Optimization Algorithms

Méthodes d'intégration et algorithmes d'optimisation accélérés

Damien Scieur (1) , Vincent Roulet (1) , Francis Bach (1) , Alexandre d'Aspremont (1)

1 SIERRA - Statistical Machine Learning and Parsimony

We show that accelerated optimization methods can be seen as particular instances of multi-step integration schemes from numerical analysis, applied to the gradient flow equation. In comparison with recent advances in this vein, the differential equation considered here is the basic gradient flow and we show that multi-step schemes allow integration of this differential equation using larger step sizes, thus intuitively explaining acceleration results.

Domaines: Optimisation et contrôle [math.OC]

Damien Scieur, Vincent Roulet, Francis Bach, Alexandre d'Aspremont. Integration Methods and Accelerated Optimization Algorithms. 2017. ⟨hal-01474045⟩