A Recurrent Variational Autoencoder for Speech Enhancement

Simon Leglaive (1, 2) , Xavier Alameda-Pineda (3) , Laurent Girin (4, 3) , Radu Horaud (3)

1 CentraleSupélec
2 IETR - Institut d'Électronique et des Technologies du numéRique
3 PERCEPTION - Interpretation and Modelling of Images and Videos
4 GIPSA-CRISSP - GIPSA - Cognitive Robotics, Interactive Systems, & Speech Processing

This paper presents a generative approach to speech enhancement based on a recurrent variational autoencoder (RVAE). The deep generative speech model is trained using clean speech signals only, and it is combined with a nonnegative matrix factorization noise model for speech enhancement. We propose a variational expectation-maximization algorithm where the encoder of the RVAE is finetuned at test time, to approximate the distribution of the latent variables given the noisy speech observations. Compared with previous approaches based on feed-forward fully-connected architectures, the proposed recurrent deep generative speech model induces a posterior temporal dynamic over the latent variables, which is shown to improve the speech enhancement results.

Mots clés: Recurrent variational autoencoders; Speech enhancement; Nonnegative matrix factorization; Variational inference

Domaines: Son [cs.SD]; Traitement du signal et de l'image [eess.SP]; Réseau de neurones [cs.NE]; Intelligence artificielle [cs.AI]

Simon Leglaive, Xavier Alameda-Pineda, Laurent Girin, Radu Horaud. A Recurrent Variational Autoencoder for Speech Enhancement. ICASSP 2020 - IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, May 2020, Barcelone (virtual), Spain. pp.371-375, ⟨10.1109/ICASSP40776.2020.9053164⟩. ⟨hal-02329000v2⟩