Dr. Martin Fischbach

Abstract: This systematic review investigates the current state of research on multimodal fusion methods, i.e., the joint analysis of multimodal inputs, for intentional, instruction-based human-computer interactions, focusing on the combination of speech and spatially expressive modalities such as gestures, touch, pen, and gaze. We examine 50 systems from a User-Centered Design perspective, categorizing them by modality combinations, fusion strategies, application domains and media, as well as reusability. Our findings highlight a predominance of descriptive late fusion methods, limited reusability, and a lack of standardized tool support, hampering rapid prototyping and broader applicability. We identify emerging trends in machine learning-based fusion and outline future research directions to advance reusable and user-centered multimodal systems.

Abstract:

Cybersickness is still a prominent risk factor potentially affecting the usability of virtual reality applications. Automated real-time detection of cybersickness promises to support a better general understanding of the phenomena and to avoid and counteract its occurrence. It could be used to facilitate application optimization, that is, to systematically link potential causes (technical development and conceptual design decisions) to cybersickness in closed-loop user-centered development cycles. In addition, it could be used to monitor, warn, and hence safeguard users against any onset of cybersickness during a virtual reality exposure, especially in healthcare applications. This article presents a novel real-time-capable cybersickness detection method by deep learning of augmented physiological data. In contrast to related preliminary work, we are exploring a unique combination of mid-immersion ground truth elicitation, an unobtrusive wireless setup, and moderate training performance requirements. We developed a proof-of-concept prototype to compare (combinations of) convolutional neural networks, long short-term memory, and support vector machines with respect to detection performance. We demonstrate that the use of a conditional generative adversarial network-based data augmentation technique increases detection performance significantly and showcase the feasibility of real-time cybersickness detection in a genuine application example. Finally, a comprehensive performance analysis demonstrates that a four-layered bidirectional long short-term memory network with the developed data augmentation delivers superior performance (91.1% F1-score) for real-time cybersickness detection. To encourage replicability and reuse in future cybersickness studies, we released the code and the dataset as publicly available.

Abstract: Multimodal Interfaces (MMIs) supporting the synergistic use of natural modalities like speech and gesture have been conceived as promising for spatial or 3D interactions, e.g., in Virtual, Augmented, and Mixed Reality (XR for short). Yet, the currently prevailing user interfaces are unimodal. Commercially available software platforms like the Unity or Unreal game engines simplify the complexity of developing XR applications through appropriate tool support. They provide ready-to-use device integration, e.g., for 3D controllers or motion tracking, and according interaction techniques such as menus, (3D) point-and-click, or even simple symbolic gestures to rapidly develop unimodal interfaces. A comparable tool support is yet missing for multimodal solutions in this and similar areas. We believe that this hinders user-centered research based on rapid prototyping of MMIs, the identification and formulation of practical design guidelines, the development of killer applications highlighting the power of MMIs, and ultimately a widespread adoption of MMIs. This article investigates potential reasons for the ongoing uncommonness of MMIs. Our case study illustrates and analyzes lessons learned during the development and application of a toolchain that supports rapid development of natural and synergistic MMIs for XR use-cases. We analyze the toolchain in terms of developer usability, development time, and MMI customization. This analysis is based on the knowledge gained in years of research and academic education. Specifically, it reflects on the development of appropriate MMI tools and their application in various demo use-cases, in user-centered research, and in the lab work of a mandatory MMI course of an HCI master’s program. The derived insights highlight successful choices made as well as potential areas for improvement.

Abstract: Objective: Gait adaptation to environmental challenges is fundamental for independent and safe community ambulation. The possibility of precisely studying gait modulation using standardized protocols of gait analysis closely resembling everyday life scenarios is still an unmet need.Methods: We have developed a fully-immersive virtual reality (VR) environment where subjects have to adjust their walking pattern to avoid collision with a virtual agent (VA) crossing their gait trajectory. We collected kinematic data of 12 healthy young subjects walking in real world (RW) and in the VR environment, both with (VR/A+) and without (VR/A-) the VA perturbation. The VR environment closely resembled the RW scenario of the gait laboratory. To ensure standardization of the obstacle presentation the starting time speed and trajectory of the VA were defined using the kinematics of the participant as detected online during each walking trial.Results: We did not observe kinematic differences between walking in RW and VR/A-, suggesting that our VR environment per se might not induce significant changes in the locomotor pattern. When facing the VA all subjects consistently reduced stride length and velocity while increasing stride duration. Trunk inclination and mediolateral trajectory deviation also facilitated avoidance of the obstacle.Conclusions: This proof-of-concept study shows that our VR/A+ paradigm effectively induced a timely gait modulation in a standardized immersive and realistic scenario. This protocol could be a powerful research tool to study gait modulation and its derangements in relation to aging and clinical conditions.

Abstract: This work introduces an interaction technique to determine the user’s non-verbal deixis in Virtual Reality (VR) applications. We tailored it for multimodal speech & gesture interfaces (MMIs). Here, non-verbal deixis is often determined by the use of ray-casting due to its simplicity and intuitiveness. However, ray-casting’s rigidness and dichotomous nature pose limitations concerning the MMI’s flexibility and efficiency. In contrast, our technique considers a more comprehensive set of directional cues to determine non-verbal deixis and provides probabilistic output to tackle these limitations. We present a machine-learning-based reference implementation of our technique in VR and the results of a first performance benchmark. Future work includes an in-depth user study evaluating our technique’s user experience in an MMI.

Abstract: Multimodal Interfaces (MMIs) combining speech and spatial input have the potential to elicit minimal cognitive load. Low cognitive load increases effectiveness as well as user satisfaction and is regarded as an important aspect of intuitive use. While this potential has been extensively theorized in the research community, experiments that provide supporting observations based on functional interfaces are still scarce. In particular, there is a lack of studies comparing the commonly used Unimodal Interfaces (UMIs) with theoretically superior synergistic MMI alternatives. Yet, these studies are an essential prerequisite for generalizing results, developing practice-oriented guidelines, and ultimately exploiting the potential of MMIs in a broader range of applications. This work contributes a novel observation towards the resolution of this shortcoming in the context of the following combination of applied interaction techniques, tasks, application domain, and technology: We present a comprehensive evaluation of a synergistic speech & touch MMI and a touch-only menu-based UMI (interaction techniques) for selection and system control tasks in a digital tabletop game (application domain) on an interactive surface (technology). Cognitive load, user experience, and intuitive use are evaluated, with the former being assessed by means of the dual-task paradigm. Our experiment shows that the implemented MMI causes significantly less cognitive load and is perceived significantly more usable and intuitive than the UMI. Based on our results, we derive recommendations for the interface design of digital tabletop games on interactive surfaces. Further, we argue that our results and design recommendations are suitable to be generalized to other application domains on interactive surfaces for selection and system control tasks.

Abstract: Joint applications of virtual reality (VR) systems and electroencephalography (EEG) offer numerous new possibilities ranging from behavioral science to therapy. VR systems allow for highly controlled experimental environments, while EEG offers a non-invasive window to brain activity with a millisecond-ranged temporal resolution. However, EEG measurements are highly susceptible to electromagnetic (EM) noise and the influence of EM noise of head-mounted-displays (HMDs) on EEG signal quality has not been conclusively investigated. In this paper, we propose a structured approach to test HMDs for EM noise potentially harmful to EEG measures. The approach verifies the impact of HMDs on the frequency- and time-domain of the EEG signal recorded in healthy subjects. The verification task includes a comparison of conditions with and without an HMD during (i) an eyes-open vs. eyes-closed task, and (ii) with respect to the sensory- evoked brain activity. The approach is developed and tested to derive potential effects of two commercial HMDs, the Oculus Rift and the HTC Vive Pro, on the quality of 64-channel EEG measurements. The results show that the HMDs consistently introduce artifacts, especially at the line hum of 50 Hz and the HMD refresh rate of 90 Hz, respectively, and their harmonics. The frequency range that is typically most important in non-invasive EEG research and applications (<50 Hz) however, remained largely unaffected. Hence, our findings demonstrate that high-quality EEG recordings, at least in the frequency range up to 50 Hz, can be obtained with the two tested HMDs. However, the number of commercially available HMDs is constantly rising. We strongly suggest to thoroughly test such devices upfront since each HMD will most likely have its own EM footprint and this article provides a structured approach to implement such tests with arbitrary devices.

Abstract: This paper introduces a method for computing the difficulty of selection tasks in virtual environments using pointing metaphors by operationalizing an established human motor behavior model. In contrast to previous work, the difficulty is calculated automatically at run-time for arbitrary environments. We present and provide the implementation of our method within Unity 3D. The difficulty is computed based on a contextual analysis of spatial boundary conditions, i.e., target object size and shape, distance to the user, and occlusion. We believe our method will enable developers to build adaptive systems that automatically equip the user with the most appropriate selection technique according to the context. Further, it provides a standard metric to better evaluate and compare different selection techniques.

Abstract: Multimodal Interfaces (MMIs) have been considered to provide promising interaction paradigms for Virtual Reality (VR) for some time. However, they are still far less common than unimodal interfaces (UMIs). This paper presents a summative user study comparing an MMI to a typical UMI for a design task in VR. We developed an application targeting creative 3D object manipulations, i.e., creating 3D objects and modifying typical object properties such as color or size. The associated open user task is based on the Torrence Tests of Creative Thinking. We compared a synergistic multimodal interface using speech-accompanied pointing/grabbing gestures with a more typical unimodal interface using a hierarchical radial menu to trigger actions on selected objects. Independent judges rated the creativity of the resulting products using the Consensual Assessment Technique. Additionally, we measured the creativity-promoting factors flow, usability, and presence. Our results show that the MMI performs on par with the UMI in all measurements despite its limited flexibility and reliability. These promising results demonstrate the technological maturity of MMIs and their potential to extend traditional interaction techniques in VR efficiently.

Abstract: Biometric measures such as the electroencephalogram (EEG) promise to become viable alternatives to subjective questionnaire ratings for the evaluation of psychophysical effects associated with Virtual Reality (VR) systems, as they provide objective and continuous measurements without breaking the exposure. The extent to which the EEG signal can be disturbed by the presence of VR sys- tems, however, has been barely investigated. This study outlines how to evaluate the compatibility of a given EEG-VR setup on the example of two commercial head-mounted displays (HMDs), the Oculus Rift and the HTC Vive Pro. We use a novel experimental protocol to compare the spectral composition between conditions with and without an HMD present during an eyes-open vs. eyes-closed task. We found general artifacts at the line hum of 50 Hz, and additional HMD refresh rate artifacts (90 Hz) for the Oculus rift exclusively. Frequency components typically most interesting to non-invasive EEG research and applications (<50 Hz), however, remained largely unaffected. We observed similar topographies of visually-induced modulation of alpha band power for both HMD conditions in all subjects. Hence, the study introduces a necessary validation test for HMDs in combination with EEG and further promotes EEG as a potential biometric measurement method for psychophysical effects in VR systems.

Abstract: Several transition techniques (TTs) exist for Virtual Reality (VR) that allow users to travel to a new target location in the vicinity of their current position. To overcome a greater distance or even move to a different Virtual Environment (VE) other TTs are required that allow for an immediate, quick, and believable change of location. Such TTs are especially relevant for VR user studies and storytelling in VR, yet their effect on the experienced presence, illusion of virtual body ownership (IVBO), and naturalness as well as their efficiency is largely unexplored. In this paper we thus identify and compare three metaphors for transitioning between VEs with respect to those qualities: an in-VR head-mounted display metaphor, a turn around metaphor, and a simulated blink metaphor. Surprisingly, the results show that the tested metaphors did not affect the experienced presence and IVBO. This is especially important for researchers and game designers who want to build more natural VEs.

Abstract: We describe a social robot acting as a game master in an interactive tabletop role-playing game. The Robot Game Master (RGM) takes on the role of different characters, which the human players meet during the adventure, as well as of the narrator. The demonstration presents a novel software and hardware platform that allows the robot to (1) proactively lead through the storyline and to (2) react to changes in the ongoing game in real-time, while (3) fostering players\u0027 collaborations.

Abstract: Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality.

Abstract: Multimodal interfaces for Virtual Reality (VR), e.g., based on speech and gesture input/output (I/O), often exhibit complex system architectures. Tight couplings between the required I/O processing stages and the underlying scene representation and the simulator system’s flow-of-control tend to result in high development and maintainability costs. This paper presents a maintainable solution for realizing such interfaces by means of a cherry-picking approach. A reusable multimodal I/O processing platform is combined with the simulation and rendering capabilities of the Unity game engine, allowing to exploit the game engine’s superior API usability and tool support. The approach is illustrated based on the development of a multimodal VR adventure game called Space Tentacles.

Abstract: Multimodal interfaces (MMIs) are a promising human-computer interaction paradigm. They are feasible for a wide rang of environments, yet they are especially suited if interactions are spatially and temporally grounded with an environment in which the user is (physically) situated. Real-time interactive systems (RISs) are technical realizations for situated interaction environments, originating from application areas like virtual reality, mixed reality, human-robot interaction, and computer games. RISs include various dedicated processing-, simulation-, and rendering subsystems which collectively maintain a real-time simulation of a coherent application state. They thus fulfil the complex functional requirements of their application areas. Two contradicting principles determine the architecture of RISs: coupling and cohesion. On the one hand, RIS subsystems commonly use specific data structures for multiple purposes to guarantee performance and rely on close semantic and temporal coupling between each other to maintain consistency. This coupling is exacerbated if the integration of artificial intelligence (AI) methods is necessary, such as for realizing MMIs. On the other hand, software qualities like reusability and modifiability call for a decoupling of subsystems and architectural elements with single well-defined purposes, i.e., high cohesion. Systems predominantly favour performance and consistency over reusability and modifiability to handle this contradiction. They thus accept low maintainability in general and hindered scientific progress in the long-term. This thesis presents six semantics-based techniques that extend the established entity-component system (ECS) pattern and pose a solution to this contradiction without sacrificing maintainability: semantic grounding, a semantic entity-component state, grounded actions, semantic queries, code from semantics, and decoupling by semantics. The extension solves the ECS pattern\u0027s runtime type deficit, improves component granularity, facilitates access to entity properties outside a subsystem\u0027s component association, incorporates a concept to semantically describe behavior as complement to the state representation, and enables compatibility even between RISs. The presented reference implementation Simulator X validates the feasibility of the six techniques and may be (re)used by other researchers due to its availability under an open-source licence. It includes a repertoire of common multimodal input processing steps that showcase the particular adequacy of the six techniques for such processing. The repertoire adds up to the integrated multimodal processing framework miPro, making Simulator X a RIS platform with explicit MMI support. The six semantics-based techniques as well as the reference implementation are validated by four expert reviews, multiple proof of concept prototypes, and two explorative studies. Informal insights gathered throughout the design and development supplement this assessment in the form of lessons learned meant to aid future development in the area.

Abstract: Modularity, modifiability, reusability, and API usability are important qualities that determine the maintainability of complex software architectures typical for Virtual, Augmented, and Mixed Reality (VR, AR, MR) applications. These architectures address various input-, output-, and processing aspects, which are usually implemented by dedicated software modules. Collectively, these modules have to maintain the real-time simulation of a coherent application state. This requirement, however, implicates multiple semantic as well as temporal state representation- and access interdependencies between modules, exacerbating maintainable solutions. This paper presents five semantics-based software techniques for state management that extend the well-established entity-component system (ECS) pattern, foster modularity and enhance overall maintainability. A walk-through of typical implementation aspects of multimodal (speech and gesture) interfaces is used to highlight the techniques\u0027 benefits, providing a typical example for demanding software architectures in VR, AR and MR. Finally, central implementation details are compared against prominent alternatives.

Abstract: This paper introduces an interactive surface concept for Mixed Reality (MR) tabletop games that combines a variable (LCD and/or projection) screen configuration with the detection of finger touches, in-air gestures, and tangibles. It is low-cost and minimally requires an ordinary table, a TV screen, and a Kinect v2 sensor. Existing applications can easily be connected by being compliant to standards. The concept is intended to foster further research on collaborative tabletop situations, not limited to games, but also in- cluding learning, meetings, and social interaction.

Abstract: This paper presents a mixed reality tabletop role-playing game with a novel combination of interaction styles and gameplay mechanics. Our contribution extends previous approaches by abandoning the traditional turn-based gameplay in favor of simultaneous real-time interaction. The increased cognitive and physical load during the simultaneous control of multiple game characters is counteracted by two features: First, certain game characters are equipped with AI-driven capabilities to become semi-autonomous virtual agents. Second, (groups of) these agents can be instructed by high-level commands via a multimodal—speech and gesture—interface.

Abstract: We present four raycast-based techniques that determine the transformation between a depth camera\u0027s coordinate system and the coordinate system defined by a rectangular surface. In addition, the surface\u0027s dimensions are measured. In contrast to other approaches, these techniques limit additional hardware requirements to commonly available, low-cost artifacts and focus on simple non-laborious procedures. A preliminary study examining our Kinect~v2-based proof of concept revealed promising first results. The utilized software is available as an open-source project.

Abstract: This poster presents a maintainable method to manage lexical information required for multimodal interfaces. It is tailored for the application in real-time interactive systems, specifically for Virtual Reality, and solves three problems commonly encountered in this context: (1) The lexical information is defined on and grounded in a common knowledge representation layer (KRL) based on OWL. The KRL describes application objects and possible system functions in one place and avoids error-prone redundant data management. (2) The KRL is tightly integrated into the simulator platform using a semantically enriched object model that is auto-generated from the KRL and thus fosters high performance access. (3) A well-defined interface provides application wide access to semantic application state information in general and the lexical information in specific, which greatly contributes to decoupling, maintainability, and reusability.

Abstract: Maintainability, i.e. reusability, modifiability, and modularity, is a critical non-functional quality requirement, especially for software frameworks. Its fulfilment is already challenging for low-interactive application areas. It is additionally complicated by complex system designs of Real-time Interactive Systems (RISs), required for Augmented, Mixed, and Virtual Reality, as well as computer games. If such systems incorporate AI methods, as required for the implementation of multimodal interfaces or smart environments, it is even further exacerbated. Existing approaches strive to establish software technical solutions to support the close temporal and semantic coupling required for multimodal processing and at the same time preserve a general decoupling principle between involved software modules. We present two key solutions that target the semantic coupling issue: (1) a semantics-based access scheme to principal elements of the application state and (2) the specification of effects by means of semantic function descriptions for multimodal processing. Both concepts are modeled in an OWL ontology. The applicability of our concepts is showcased by a prototypical implementation and explained by an interaction example that is applied for two application areas.

Abstract: Multimodal interaction frameworks are an efficient means of utilizing many existing processing and fusion techniques in a wide variety of application areas, even by non-experts. However, the application of these frameworks to highly interactive application areas like VR, AR, MR, and computer games in a reusable, modifiable, and modular manner is not straightforward. It currently lacks some software technical solutions that (1) preserve the general decoupling principle of platforms and at the same time (2) provide the required close temporal as well as semantic coupling of involved software modules and multimodal processing steps. This thesis approches current challenges and aims at providing the research community with a framework that fosters repeatability of scientific achievements and the ability to built on previous results.

Abstract: This article describes four software techniques to enhance the overall quality of multimodal processing software and to include concurrency and variance due to individual characteristics and cultural context. First, the processing steps are decentralized and distributed using the actor model. Second, functor objects decouple domain- and application-specific operations from universal processing methods. Third, domain specific languages are provided inside of specialized feature processing units to define necessary algorithms in a human-readable and comprehensible format. Fourth, constituents of the DSLs (including the functors) are semantically grounded into a common ontology supporting syntactic and semantic correctness checks as well as code-generation capabilities. These techniques provide scalable, customizable, and reusable technical solutions for reoccurring multimodal processing tasks.

Abstract: Quest - XRoads is a multimodal and multimedia mixed reality version of the traditional role-play tabletop game Quest: Zeit der Helden. The original game concept is augmented with virtual content, controllable via auditory, tangible and spatial interfaces to permit a novel gaming experience and to increase the satisfaction while playing. The demonstration consists of a turn-based skirmish, where up to four players have to collaborate to defeat an opposing player. In order to be victorious, players have to control heroes or villains and use their abilities via speech, gesture, touch as well as tangible interactions.

Abstract: Localisation, i.e. determining the position of the user(s) or devices, constitutes the key requirement for almost all types of mobile pervasive games. However, current marker-based localisation systems, e.g.\ based on QR Markers, present drawbacks that limit game deployment, scalability and maintainability. In this paper, we propose an alternative to solve these issues and introduce the first steps towards its full realisation. Our approach relies on markerless picture matching using a natural feature detection algorithm. Players reproduce a camera shot of a real-world site in order to confirm their presence, and progress further in the game. One of the game critical requirements is to provide accurate recognition while preserving application responsiveness with a large range of mobile devices and camera resolutions. We developed a proof-of-concept system and determined the best picture resolutions and feature numbers necessary to preserve both accurracy and responsiveness on diverse mobile devices. Our first results demonstrate the feasibility to achieve precise recognition within real-time constraints. We believe such localisation system have the potential to considerably facilitate pervasive game authoring while promoting new type of game mechanics.

Abstract: Dieser Artikel beschreibt die digitale Umsetzung eines rollenspielbasierten Brettspiels zur Exploration neuer Interaktionstechniken. Als gemeinsame Mixed-Reality-Spielumgebung dient ein Multitouch-Tisch mit Objekterkennung für haptisch erfassbare Spielelemente (Spielfiguren, Karten, ...). Das System ergänzt die realen Objekte mit multimedialen Informationen gemäß des aktuellen Spielgeschehens. Die Integration tragbarer Endgeräte über eine HTML5 -Schnittstelle ermöglicht private und individualisierte Interaktionsbereiche. Das System vereint unterschiedliche Interaktionstechniken wie Touch-Eingabe und Interaktion mit greifbaren Objekten, um den Zufriedenheitsgrad bei Interaktionen positiv zu beeinflussen. Eine Pilotstudie mit rollenspielerfahrenen Benutzern prüft die Akzeptanz der neuen Spiel- und Interaktionsmöglichkeiten.

Abstract: This article describes a benchmark for evaluating adequacy of input devices for bimanual direct interaction techniques typically found in VR/AR applications. The benchmark implements a puzzle-like scenario inspired from shape and color sorting games, in which the user manipulates the position, rotation and scale of 3D spheres. The continuous interactions are combined with multimodal state-change actions using either a button or speech interface. A follow-up usability study utilizes the benchmark to evaluate the performance of one professional and one consumer-grade tracking system for both state-changing interfaces. The results reveal similar adequacy for both tracking systems under both multimodal conditions.

Abstract: Recent developments in the field of semi-immersive display technologies pro- vide new possibilities for engaging users in interactive three-dimensional virtual environments (VEs). For instance, combining low-cost tracking systems (such as the Microsoft Kinect) and multi-touch interfaces enables inexpensive and easily maintainable interactive setups. The goal of this work is to bring together virtual as well as real objects on a stereoscopic multi- touch enabled tabletop surface. Therefore, we present a prototypical implementation of such a mixed reality (MR) space for tangible interaction by extending the smARTbox FLBS12. The smARTbox is a responsive touch-enabled stereoscopic out-of-the-box system that is able to track users and objects above as well as on the surface. We describe the prototypical hard- and software setup which extends this setup to a MR space, and highlight design challenges for the several application examples.

Abstract: This paper presents an enhanced version of a portable out-of-the-box platform for semi-immersive interactive applications. The enhanced version combines stereoscopic visualization, markerless user tracking, and multi-touch with self-reflection of users and tangible object inter- action. A virtual fish tank simulation demonstrates how real and virtual worlds are seamlessly blended by providing a multi-modal interaction ex- perience that utilizes a user-centric projection, body, and object tracking, as well as a consistent integration of physical and virtual properties like appearance and causality into a mixed real/virtual world.

Abstract: This article evaluates the utility of three technical design approaches implemented during the development of a Realtime Interactive Systems (RIS) architecture focusing on the areas of Virtual and Augmented Reality (VR and AR), Robotics, and Human-Computer Interaction (HCI). The design decisions are (1) the choice of the Scala programming language, (2) the implementation of the actor computational model, and (3) the central incorporation of ontologies as a base for semantic modeling, required for several Artificial Intelligence (AI) methods. A white-box expert review is applied to a detailed use case illustrating an interactive and multimodal game scenario, which requires a number of complex functional features like speech and gesture processing and instruction mapping. The review matches the three design decisions against three comprehensive non-functional requirements from software engineering: Reusability, scalability, and extensibility. The qualitative evaluation is condensed to a semi-quantitative summary, pointing out the benefits of the chosen technical design.

Abstract: Interaktive Medien mit räumlicher Darstellung virtueller Inhalte finden verstärkt Einzug in verschiedene Anwendungsfelder, was zu einer stetig wachsenden Nachfrage für kostengünstige Verfahren zur Bestimmung der Kopfposition eines Betrachters führt. In diesem Beitrag evaluieren wir zwei Headtrackingverfahren in interaktiven virtuellen Umgebungen auf Basis der Microsoft Kinect. Wir vergleichen die beiden Verfahren mit einem professionellen optischen Trackingsystem und zeigen Vor- und Nachteile auf.

Abstract: This paper presents a semi-immersive, multimodal fish tank simulation realized using the smARTbox, an out-of-the-box platform for intelligent interactive applications. The smARTbox provides portability, stereoscopic visualization, marker-less user tracking and direct interscopic touch input. Off-the-shelf hardware is combined with a state-of-the-art simulation platform to assemble a powerful system environment, which facilitates direct (touch) and indirect (movement) interaction.

Abstract: Recent developments in the �elds of interactive display technologies provide new possibilities for engaging visitors in interactive three-dimensional virtual art exhibitions. Tracking and interaction technologies such as the Microsoft Kinect and emerging multi-touch interfaces enable inexpensive and low-maintenance interactive art setups while providing portable solutions for engaging presentations and exhibitions. In this paper we describe the smARTbox, which is a responsive touch-enabled stereoscopic out-of-the-box technology for interactive art setups. Based on the described technologies, we sketch an interactive semi-immersive virtual �sh tank implementation that enables direct and indirect interaction with visitors.

Abstract: We present SiXton’s Curse – a computer game – to illustrate the benefits of a novel simulation platform. Simulator X targets virtual, augmented, and mixed reality applications as well as computer games. The game simulates a medieval village called SiXton that can be explored and experienced using gestures and speech for input. SiXton’s Curse utilizes multiple independent components for physical simulation, sound and graphics rendering, artificial intelligence, as well as for multi-modal interaction (MMI). The components are already an integral part of Simulator X’s current version. Building on Hewitt’s actor model, the Simulator X platform enables the developer to easily exploit the capabilities of modern hardware architectures. A state variable concept is implemented on top of the actor model to grant uniform and easy access to global states and values by using the internal mechanisms of the actor model. Communication via an asynchronous messaging interface reduces component coupling. The scalability of the actor model provides a uniform concurrency paradigm on different levels of granularity as well as exchangeability of architectural elements and components.