Learning from Video and Text via Large-Scale Discriminative Clustering

Antoine Miech (1, 2, 3) , Jean-Baptiste Alayrac (1, 2, 3) , Piotr Bojanowski (2) , Ivan Laptev (1, 2, 3) , Josef Sivic (1, 2, 3, 4)

1 PSL - Université Paris Sciences et Lettres
2 WILLOW - Models of visual object recognition and scene understanding
3 DI-ENS - Département d'informatique - ENS Paris
4 CTU - Czech Technical University in Prague

Discriminative clustering has been successfully applied to a number of weakly-supervised learning tasks. Such applications include person and action recognition, text-to-video alignment, object co-segmentation and co-localization in videos and images. One drawback of dis-criminative clustering, however, is its limited scalability. We address this issue and propose an online optimization algorithm based on the Block-Coordinate Frank-Wolfe algorithm. We apply it to the problem of weakly-supervised learning of actions and actors from movies and corresponding movie scripts. The scaling up of the learning problem to 66 feature-length movies enables us to significantly improve weakly-supervised action recognition.

Domaines: Informatique [cs]; Intelligence artificielle [cs.AI]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Antoine Miech, Jean-Baptiste Alayrac, Piotr Bojanowski, Ivan Laptev, Josef Sivic (Dir.). Learning from Video and Text via Large-Scale Discriminative Clustering. published by the authors, 2017. ⟨hal-01569540v1⟩