Constraints on the Use of Language, Gesture and Speech for Multimodal Dialogues

Bertrand Gaiffe (1) , Laurent Romary (1)

1 LANGUE ET DIALOGUE - Human-machine dialogue with a significant language component

We try to show how language, gesture and perception can be seen in a uniform way from the perspective of referential analysis, by looking at the specific constraints which underly the speaker's choice of a given expression. To this end, we first situate the relative importance of speech and gesture in man-machine communication. Then, we concentrate upon the specific effects resulting from the combination of verbal, gestural and perceptual information, showing that on the one hand the three provide structural constraints to the objects which are being referred to and on the other hand that any referring operation, whatever its origin, has to be interpreted within a localized frame, with some consequences upon dialogue management.

Domaines: Informatique et langage [cs.CL]

Bertrand Gaiffe, Laurent Romary. Constraints on the Use of Language, Gesture and Speech for Multimodal Dialogues. Constraints On The Use Of Language Gesture And Speech For Multimodal Dialogues, ACL Special Interest Group on Multimedia Language Processing (SIGMEDIA), Jul 1997, Madrid, Spain. pp.94-98. ⟨hal-00757452⟩