A multi-modal perception based assistive robotic system for the elderly

Christophe Mollaret (1, 2) , Alhayat Ali Mekonnen (1) , Frédéric Lerasle (1) , Isabelle Ferrané (2) , Julien Pinquier (2) , Blandine Boudet (3) , Pierre Rumeau (3)

1 LAAS-RAP - Équipe Robotique, Action et Perception
2 IRIT-SAMoVA - Équipe Structuration, Analyse et MOdélisation de documents Vidéo et Audio
3 Gérontopôle

In this paper, we present a multi-modal perception based framework to realize a non-intrusive domestic assistive robotic system. It is non-intrusive in that it only starts interaction with a user when it detects the user's intention to do so. All the robot's actions are based on multi-modal perceptions which include user detection based on RGB-D data, user's intention-for-interaction detection with RGB-D and audio data, and communication via user distance mediated speech recognition. The utilization of multi-modal cues in different parts of the robotic activity paves the way to successful robotic runs (94% success rate). Each presented perceptual component is systematically evaluated using appropriate dataset and evaluation metrics. Finally the complete system is fully integrated on the PR2 robotic platform and validated through system sanity check runs and user studies with the help of 17 volunteer elderly participants.

Mots clés: Robotic Perception; Human-Robot Interaction; Assistive Technology; Elderly Care; Intention Detection; Multi-modal Data Fusion

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Robotique [cs.RO]

Christophe Mollaret, Alhayat Ali Mekonnen, Frédéric Lerasle, Isabelle Ferrané, Julien Pinquier, et al.. A multi-modal perception based assistive robotic system for the elderly. Computer Vision and Image Understanding, 2016, Special issue on Assistive Computer Vision and Robotics : Assistive Solutions for Mobility, Communication and HMI, 149, pp.78-97. ⟨10.1016/j.cviu.2016.03.003⟩. ⟨hal-01300463⟩