Strategies for multiple feature fusion with Hierarchical HMM: Application to activity recognition from wearable audiovisual sensors

Julien Pinquier (1) , Svebor Karaman (2) , Laetitia Letoupin (2) , Patrice Guyot (1) , Rémi Mégret (3) , Jenny Benois-Pineau (2) , Yann Gaëstel (4) , Jean François Dartigues (5)

1 IRIT-SAMoVA - Équipe Structuration, Analyse et MOdélisation de documents Vidéo et Audio
2 LaBRI - Laboratoire Bordelais de Recherche en Informatique
3 IMS - Laboratoire de l'intégration, du matériau au système
4 ISPED - Institut de Santé Publique, d'Epidémiologie et de Développement
5 Epidémiologie et Biostatistique [Bordeaux]

In this paper, we further develop the research on recognition of activities, in videos recorded with wearable cameras, with Hierarchical Hidden Markov Model classifiers. The visual scenes being of a strong complexity in terms of motion and visual content, good performances have been obtained using multiple visual and audio cues. The adequate fusion of features from physically different description spaces remains an open issue not only for this particular task, but in multiple problems of pattern recognition. A study of optimal fusion strategies in the HMM framework is proposed. We design and exploit early, intermediate and late fusions with emitting states in the H-HMM. The results obtained on a corpus recorded by healthy volunteers and patients in a longitudinal dementia study allow choosing optimal fusion strategies as a function of target activity.

Domaines: Multimédia [cs.MM]

Julien Pinquier, Svebor Karaman, Laetitia Letoupin, Patrice Guyot, Rémi Mégret, et al.. Strategies for multiple feature fusion with Hierarchical HMM: Application to activity recognition from wearable audiovisual sensors. 21st International Conference on Pattern Recognition (ICPR 2012), Nov 2012, Tsukuba, Japan. pp.3192 - 3195. ⟨hal-00853854⟩