Predicting Actions from Static Scenes

Tuan-Hung Vu (1) , Catherine Olsson (2) , Ivan Laptev (1) , Aude Oliva (2) , Josef Sivic (1)

1 WILLOW - Models of visual object recognition and scene understanding
2 CSAIL - Computer Science and Artificial Intelligence Laboratory [Cambridge]

Human actions naturally co-occur with scenes. In this work we aim to discover action-scene correlation for a large number of scene categories and to use such correlation for action prediction. Towards this goal, we collect a new SUN Action dataset with manual annotations of typical human actions for 397 scenes. We next discover action-scene associations and demonstrate that scene categories can be well identified from their associated actions. Using discovered associations, we address a new task of predicting human actions for images of static scenes. We evaluate prediction of 23 and 38 action classes for images of indoor and outdoor scenes respectively and show promising results. We also propose a new application of geo-localized action prediction and demonstrate ability of our method to automatically answer queries such as "Where is a good place for a picnic?" or "Can I cycle along this path?".

Mots clés: scene recognition; functional properties; Action prediction

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Tuan-Hung Vu, Catherine Olsson, Ivan Laptev, Aude Oliva, Josef Sivic. Predicting Actions from Static Scenes. ECCV'14 - 13th European Conference on Computer Vision, Sep 2014, Zurich, Switzerland. pp.421-436, ⟨10.1007/978-3-319-10602-1_28⟩. ⟨hal-01053935⟩