Multiview approaches to event detection and scene analysis

Slim Essid (1, 2) , Sanjeel Parekh (3) , Ngoc Q. K. Duong (3) , Romain Serizel (4, 1, 2) , Alexey Ozerov (3) , Fabio Antonacci (5) , Augusto Sarti (5)

1 S2A - Signal, Statistique et Apprentissage
2 IDS - Département Images, Données, Signal
3 Technicolor R & I [Cesson Sévigné]
4 MULTISPEECH - Speech Modeling for Facilitating Oral-Based Communication
5 Dipartimento di Elettronica e Informazione

This chapter addresses sound scene and event classification in multiview settings, that is, settings where the observations are obtained from multiple sensors, each sensor contributing a particular view of the data (e.g., audio microphones, video cameras, etc.). We briefly introduce some of the techniques that can be exploited to effectively combine the data conveyed by the different views under analysis for a better interpretation. We first provide a high-level presentation of generic methods that are particularly relevant in the context of multiview and multimodal sound scene analysis. Then, we more specifically present a selection of techniques used for audiovisual event detection and microphone array-based scene analysis.

Mots clés: Multimodal scene analysis; Tensor factorization; Joint audiovisual scene analysis; Representation learning; Multichannel audio; Wiener filtering; Data fusion; Multichannel; Audio source separation; Beamforming; Matrix factorization; Multiview scene analysis; Audio source localization and tracking

Domaines: Traitement du signal et de l'image [eess.SP]; Son [cs.SD]; Apprentissage [cs.LG]

Slim Essid, Sanjeel Parekh, Ngoc Q. K. Duong, Romain Serizel, Alexey Ozerov, et al.. Multiview approaches to event detection and scene analysis. Tuomas Virtanen; Mark D. Plumbley; Dan Ellis. Computational Analysis of Sound Scenes and Events, Springer, pp.243-276, 2017, 978-3319634494. ⟨10.1007/978-3-319-63450-0_9⟩. ⟨hal-01620341⟩