After presenting my research approach, I will present a design space for multimodal interaction. The considerations involved in this design space are not only based on current technological capabilities but also on the psychological consequences of the design features on users. Within the vast world of possibilities for input modalities (from the user to the system) as well as for outputs (from the system to the user), we distinguish two types of modalities: the active and passive (implicit) modalities. For inputs, active modalities are used by the user to issue a command to the computer (e.g., a voice command or a gesture recognized by a camera). Passive modalities refer to information that is not explicitly expressed by the user, but automatically captured for enhancing the execution of a task (as in perceptual user interfaces). For example, in the “Put that there” seminal multimodal demonstrator of R. Bolt (MIT, 1980), eye tracking was used for detecting which object on screen the user is looking at. With the results of this work in hand (modality definition and design space), I then examine the software development of multimodal interaction by focusing on software architecture models and fusion mechanisms.
I therefore organize the " multimodal interaction" tutorial along the steps of a software life-cycle, by first addressing the issues of ergonomic design and then software design and development.
Keywords
Human Computer Interaction, Interaction Modality, Multimodality, Ubliquitous Computing, Augmented Reality, Software architecture, Fusion mechanisms.