Multimodal interfaces (MMIs) are a promising alternative human-computer interaction paradigm. They are feasible for a wide rang of environments, yet they are especially suited if interactions are spatially and temporally grounded with an environment in which the user is (physically) situated, like virtual reality, mixed reality, human-robot interaction, and computer games.
We use a concurrent Augmented Transition Network (cATN) to implement multimodal interfaces for a broad spectrum of demonstrations and research. It is a successor of the temporal Augmented Transition Network and is implemented in Simulator X.
The Quest V2 prototype is a mixed reality tabletop role-playing game with a novel combination of interaction styles and gameplay mechanics.
For Developers
If you are interested in using the cATN for research or teaching, then contact us. The latest version of our cATN is hosted on our institute’s GitLab server together with additional material that we use for teaching, such as how-tos and practical exercises. Getting access is as easy as sending your type of interest as well as a mail address for registration to one of the primary contact persons linked at bottom of this page.
This year's joint contribution of the HCI chair and the Psychological Ergonomics chair 'Finally on Par?! Multimodal and Unimodal Interaction for Open Creative Design Tasks in Virtual Reality' was nominated for Best Paper. Congratulations to Erik, Sara, Chris, Martin, Jean-Luc, and Marc!
This year's joint contribution of the HCI chair and the Psychological Ergonomics chair 'Paint that object yellow: Multimodal Interaction to Enhance Creativity During Design Tasks in VR' was awarded as Best Paper Runner-Up. Congratulations to Erik, Sara, Chris, Jean-Luc, and Marc!
Here you find Japanese descriptions of our research demos for the German-Japanese Spring School on Human Factors 2018 in collaboration of Psychological Ergonomics and HCI.
Martin Fischbach will present an extended exposé of his PhD thesis on software techniques for multimodal input processing in Realtime Interactive Systems at the International Conference on Multimodal Interaction in Seattle.
Recording, playback, and analysis of human performance in virtual environments (VEs) is an important foundation facilitating systems that respond appropriately to (un)intentional non-verbal human actions. This project targets a general solution for recording time series of entity property changes within Unity and its application to generate artificial training data for image-based human pose detection as well as to the machine-learning-supported recognition of non-verbal human object references in a VE.
The goal of this HCI thesis is to investigate and compare a novel interaction technique for determining a user's nonverbal deixis with ray-casting in the context of a multimodal speech and gesture interface in VR.
This thesis proposes a testbed for comparative evaluations of fusion engines and compares two different fusion approaches as proof of concept: temporal augmented transition networks vs. unification.
Tracking systems usually extract the position of body joints and pass them on to a final decision unit which classifies the data as different gestures. These units can be manually specified templates.
