Natural Potining for Multimodal Interfaces

Assigned Human-Computer Interaction Master Internship No Affiliation

Motivation

Multimodal Interfaces (MMIs) play a crucial role in Virtual Reality (VR) environments, enhancing user experience by enabling more intuitive, natural, and immersive interactions. Among various input modality combinations for intentional MMIs, speech and gesture input are considered the most prominent and powerful combination due to their complementarity and strong interdependence.

Providing appropriately pre-processed data for the joint analysis of multimodal inputs (multimodal fusion) is relatively easy to implement for speech - e.g. by using one of the numerous available and suitable automatic speech recognition (ASR) tools as a subsystem.

In contrast, preprocessing gestures poses a greater challenge.

The challenge is often bypassed in favor of simpler implementations, such as the Raycasting technique.

However, this results in notable limitations, including challenges with precision and an inability to interpret nuanced gestures accurately.

Scope of the Internship

These constraints highlight the need for exploring more advanced and flexible interaction methods to enhance user experience in VR environments. In previous work by the research group in the chair of HCI (Stingl, Zimmerer, Fischbach, & Latoschik, 2022), a more natural way of interaction has been introduced that considers a more comprehensive set of directional cues to determine non-verbal deixis and provides probabilistic output to tackle the drawbacks of Raycasting.

The aim of this internship is to familiarize with the provided implementation in order to further develop it into a reusable Unity package. This package should be suitable for import into other Unity applications.

As a basis for potential follow-up work, limitations and potential improvements shall then be analyzed and documented, such as the quality of training data, the training data collection process, the utilised machine learning model, and Unity integration.

Tasks and Planned Progression

1. Inspection of the technical and conceptual state of the reference work
2. Set up and test of the reference implementation in the HCI laboratory
3. Upgrade of the Unity project to a new Unity LTS version and new HMD
4. Migration of the implementation to a Unity package (and integration into the Reality Stack package server)
5. Analysis of limitations and potential for improvements
6. Documentation of the work

Insights into the Work at the HCI Chair

In addition to providing insight into the research of the MMI systems, the internship is also intended to provide a general impression of the academic work at the HCI chair. This includes regular participation in the weekly research meeting as well as participation in organizational tasks, such as the “Asset Maintenance” working group, which is responsible for the maintenance of hardware at the chair.

Schedule

References

Stingl, R., Zimmerer, C., Fischbach, M., & Latoschik, M. E. (2022). Are you referring to me? Giving virtual objects awareness. 2022 IEEE International Symposium on Mixed and Augmented Reality

Contact Persons at the University Würzburg