HMM-based Automatic Visual Speech Segmentation Using Facial Data

Utpala Musti (1) , Asterios Toutios (1) , Slim Ouni (1) , Vincent Colotte (1) , Brigitte Wrobel-Dautcourt (2) , Marie-Odile Berger (2)

1 PAROLE - Analysis, perception and recognition of speech
2 MAGRIT - Visual Augmentation of Complex Environments

We describe automatic visual speech segmentation using facial data captured by a stereo-vision technique. The segmentation is performed using an HMM-based forced alignment mechanism widely used in automatic speech recognition. The idea is based on the assumption that using visual speech data alone for the training might capture the uniqueness in the facial compo- nent of speech articulation, asynchrony (time lags) in visual and acoustic speech segments and significant coarticulation effects. This should provide valuable information that helps to show the extent to which a phoneme may affect surrounding phonemes visually. This should provide information valuable in labeling the visual speech segments based on dominant coarticulatory contexts.

Mots clés: facial speech; speech segmentation; forced alignment; coarticulation

Domaines: Autre [q-bio.OT]; Multimédia [cs.MM]; Synthèse d'image et réalité virtuelle [cs.GR]; Traitement du signal et de l'image [eess.SP]; Traitement du signal et de l'image [eess.SP]

Utpala Musti, Asterios Toutios, Slim Ouni, Vincent Colotte, Brigitte Wrobel-Dautcourt, et al.. HMM-based Automatic Visual Speech Segmentation Using Facial Data. Interspeech 2010, ISCA, Sep 2010, Makuhari, Chiba, Japan. pp.1401-1404. ⟨inria-00526776⟩