PAC-Bayesian Contrastive Unsupervised Representation Learning

Kento Nozawa (1, 2) , Pascal Germain (3) , Benjamin Guedj (4, 5, 6, 3, 7)

1 Tokyo University of Science [Tokyo]
2 RIKEN AIP - RIKEN Center for Advanced Intelligence Project [Tokyo]
3 MODAL - MOdel for Data Analysis and Learning
4 UCL - University College of London [London]
5 UCL-CS - Department of Computer science [University College of London]
6 Inria-CWI - Inria-CWI
7 Inria-London - The Inria London Programme

Contrastive unsupervised representation learning (CURL) is the state-of-the-art technique to learn representations (as a set of features) from unlabelled data. While CURL has collected several empirical successes recently, theoretical understanding of its performance was still missing. In a recent work, Arora et al. (2019) provide the first generalisation bounds for CURL, relying on a Rademacher complexity. We extend their framework to the flexible PAC-Bayes setting, allowing to deal with the non-iid setting. We present PAC-Bayesian generalisation bounds for CURL, which are then used to derive a new representation learning algorithm. Numerical experiments on real-life datasets illustrate that our algorithm achieves competitive accuracy, and yields generalisation bounds with non-vacuous values.

Domaines: Machine Learning [stat.ML]; Apprentissage [cs.LG]; Théorie [stat.TH]

Kento Nozawa, Pascal Germain, Benjamin Guedj. PAC-Bayesian Contrastive Unsupervised Representation Learning. UAI 2020 - Conference on Uncertainty in Artificial Intelligence, Aug 2020, Toronto, Canada. ⟨hal-02401282⟩