Learning Interactions and Relationships between Movie Characters

Anna Kukleva (1) , Makarand Tapaswi (2) , Ivan Laptev (2)

1 Max Planck Institute for Informatics [Saarbrücken]
2 WILLOW - Models of visual object recognition and scene understanding

Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]

Anna Kukleva, Makarand Tapaswi, Ivan Laptev. Learning Interactions and Relationships between Movie Characters. CVPR 2020- IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2020, Virtual, United States. ⟨10.1109/CVPR42600.2020.00987⟩. ⟨hal-03017606⟩