Progressive Perceptual Audio Rendering of Complex Scenes

Thomas Moeck (1, 2) , Nicolas Bonneel (1) , Nicolas Tsingos (1) , George Drettakis (1) , Isabelle Viaud-Delmon (3) , David Aloza (4)

1 REVES - Rendering and virtual environments with sound
2 Department of Computer Science [Erlangen]
3 VAP - Vulnérabilité Adaptation et Psychopathologie
4 EDEN Games

Despite recent advances, including sound source clustering and perceptual auditory masking, high quality rendering of complex virtual scenes with thousands of sound sources remains a challenge. Two major bottlenecks appear as the scene complexity increases: the cost of clustering itself, and the cost of pre-mixing source signals within each cluster. In this paper, we first propose an improved hierarchical clustering algorithm that remains efficient for large numbers of sources and clusters while providing progressive refinement capabilities. We then present a lossy pre-mixing method based on a progressive representation of the input audio signals and the perceptual importance of each sound source. Our quality evaluation user tests indicate that the recently introduced audio saliency map is inappropriate for this task. Consequently we propose a "pinnacle", loudness-based metric, which gives the best results for a variety of target computing budgets. We also performed a perceptual pilot study which indicates that in audio-visual environments, it is better to allocate more clusters to visible sound sources. We propose a new clustering metric using this result. As a result of these three solutions, our system can provide high quality rendering of thousands of 3D-sound sources on a "gamer-style" PC.

Mots clés: Audio rendering; auditory masking; ventriloquism; clustering

Domaines: Synthèse d'image et réalité virtuelle [cs.GR]

Thomas Moeck, Nicolas Bonneel, Nicolas Tsingos, George Drettakis, Isabelle Viaud-Delmon, et al.. Progressive Perceptual Audio Rendering of Complex Scenes. Symposium on Interactive 3D graphics and games (I3D 2007), ACM SIGGRAPH, Apr 2007, Seattle, United States. pp.189-196, ⟨10.1145/1230100.1230133⟩. ⟨inria-00606801⟩