An Inertial Newton Algorithm for Deep Learning

Camille Castera (1) , Jérôme Bolte (2) , Cédric Févotte (1) , Edouard Pauwels (3)

1 UT - Université de Toulouse
2 UT Capitole - Université Toulouse Capitole
3 UT3 - Université Toulouse III - Paul Sabatier

We introduce a new second-order inertial method for machine learning called INDIAN, exploiting the geometry of the loss function while only requiring stochastic approximations of the function values and the generalized gradients. This makes the method fully implementable and adapted to large-scale optimization problems such as the training of a deep neural network. The algorithm combines both gradient-descent and Newton-like behaviors as well as inertia. We prove the convergence of INDIAN to critical points for most deep learning problems. To do so, we provide a well-suited framework to analyze deep learning losses involving tame optimization in which we study the continuous dynamical system together with the discrete stochastic approximations. On the theoretical side, we also prove a sublinear convergence rate for the continuous time differential inclusion which underlies the algorithm. From an empirical point of view the algorithm shows promising results on popular DNN training benchmark problems.

Mots clés: Stochastic Optimization; Deep Learning Algorithms; Nonconvex optimization

Mots clés: Optimisation non convexe; Optimisation Stochastique; Deep Learning; Algorithmes pour le deep learning

Domaines: Apprentissage [cs.LG]; Optimisation et contrôle [math.OC]; Machine Learning [stat.ML]

Camille Castera, Jérôme Bolte, Cédric Févotte, Edouard Pauwels. An Inertial Newton Algorithm for Deep Learning. 2019. ⟨hal-02140748v3⟩