Stable recovery of the factors from a deep matrix product and application to convolutional networks. Focus on sparsity constraints

François Malgouyres (1) , Joseph Landsberg (2)

1 IMT - Institut de Mathématiques de Toulouse UMR5219
2 TAMU - Department of Mathematics [Texas]

We study a deep matrix factorization problem. It takes as input a matrix X obtained by multiplying K matrices (called factors). Each factor is obtained by applying a fixed linear operator to a vector of parameters satisfying a sparsity constraint. We provide sharp conditions on the structure of the model that guarantee the stable recovery of the factors from the knowledge of X and the model for the factors. This is crucial in order to interpret the factors and the intermediate features obtained when applying a few factors to a datum. When K = 1: the paper provides compressed sensing statements; K = 2 covers (for instance) Non-negative Matrix Factorization, Dictionary learning, low rank approximation, phase recovery. The particularity of this paper is to extend the study to deep problems. As an illustration, we detail the analysis and provide (entirely computable) guarantees for the stable recovery of a (non-neural) sparse convolutional network.

Domaines: Optimisation et contrôle [math.OC]; Statistiques [math.ST]; Apprentissage [cs.LG]

François Malgouyres, Joseph Landsberg. Stable recovery of the factors from a deep matrix product and application to convolutional networks. Focus on sparsity constraints. 2017. ⟨hal-01526083v1⟩