ISIR, University Pierre and Marie Curie

In human-to-human communication, signals from multiple channels are at work
. We communicate not only through words but also by intonation, gaze, hand and body gestures and facial expressions.
These verbal and nonverbal signals have a role in the communicative process
. They add/modify/substitute information in discourse and are highly linked with eachother.
The ease and robustness of human-human communication is due to extremely high recognition accuracy using multiple input channels and the redundant and complimentary use of several modalities
. Human computer interaction can benefit from modeling several modalities in analogous ways.
Multimodal systems represent and manipulate information from different human communication channels at multiple levels of abstraction
. The purpose of this chapter is to present current work towards establishing standards and common resources for multimodal systems.
The chapter covers an overview and survey of current multimodal systems and claries the basic terminology
It also provides recommendations on the different components of such systems.
More particularly this chapter focuses on multimodal systems that have speech either as input or output modality
The chapter discusses aspects of the various types of multimodal systems according to what speech and nonspeech modalities speech input and output is associated with
It describes systems that combine speech input with information from the visual channel face detection face recognition tracking of facial features and lipreading
systems using talking faces and conversational agents combination of speech with visual output are also presented.
Another object of attention of the chapter is on the concepts and issues related to multimodal systems combining speech input with other input modalities as well as to systems combining speech output.