Distributed Learning with Sparse Communications by Identification

Dmitry Grishchenko (1) , Franck Iutzeler (1) , Jérôme Malick (1) , Massih-Reza Amini (2)

1 DAO - Données, Apprentissage et Optimisation
2 LIG - Laboratoire d'Informatique de Grenoble

In distributed optimization for large-scale learning, a major performance limitation comes from the communications between the different entities. When computations are performed by workers on local data while a coordinator machine coordinates their updates to minimize a global loss, we present an asynchronous optimization algorithm that efficiently reduces the communications between the coordinator and workers. This reduction comes from a random sparsification of the local updates. We show that this algorithm converges linearly in the strongly convex case and also identifies optimal strongly sparse solutions. We further exploit this identification to propose an automatic dimension reduction, aptly sparsifying all exchanges between coordinator and workers.

Mots clés: optimization; asynchronous algorithms; structure-promoting regularization; proximal algorithms

Domaines: Optimisation et contrôle [math.OC]; Calcul parallèle, distribué et partagé [cs.DC]

Dmitry Grishchenko, Franck Iutzeler, Jérôme Malick, Massih-Reza Amini. Distributed Learning with Sparse Communications by Identification. SIAM Journal on Mathematics of Data Science, 2021, 3 (2), pp.715-735. ⟨10.1137/20M1347772⟩. ⟨hal-01950120v2⟩