Gesture Detection and Stroke Correction in 3D Immersive Sketching

Open Human-Computer Interaction Master Project Thesis No Affiliation

Background

Immersive design systems such as Tilt Brush, Open Brush or Gravity Sketch allow users to create 3D persistent sketches in the air around themselves using only hand gestures. Precisely sketching intended strokes in mid-air in all three dimensions is demanding (Keefe et al. (2007), Wiese et al. (2010), Monty et al. (2024)). Bio-mechanical constraints of the hand, wrist and arm contribute to fatigue and increased inaccuracy in unsupported VR sketching (Arora et al. (2017)). The natural ergonomic limitations of the human arm, shoulder and wrist lead to unintentionally curved strokes. Sketch quality diminishes, users are less efficient, and spend less time actually sketching than with traditional sketching tools (Yang & Lee, (2020), Oti & Crilly, (2021), Alex et al. (2021)).

Goal

The aim of this project is to develop a computational model to detect specific stroke gestures and correct the natural curvature of strokes drawn in mid-air. We will work with a real-time, immersive freehand sketching system that maps mid-air hand gestures to 3D mid-air ink strokes. Using either a statistical or a machine learning based approach, we will develop a computational model to correct stroke curvature inacurracies in real-time. We will collect gesture data reflecting strokes that are intended to be straight and robustly curved. We will extract key features such as arm pose, acceleration and velocity, stroke curvature, and angle deviation, and use this data to build a stroke correction model. We will then conduct a user study to validate the model. We wish to address the following broad research question: What is the influence of stroke curvature correction on the accuracy and quality of sketches, as well as on user efficiency and system usability.

Tasks

The project will focus on the following tasks:

• Acquaintance with the topic and the theoretical background
• Design and implementation of a stroke correction tool
• Comparative user study to evaluate the model
• Evaluation and presentation of results

Prerequisites

• Game engine experience (Unity)
• Familiarity with implementing VR applications
• Knowledge of research methods and statistics
• Interest in combining technical development with empirical work
• Interest in developing and improving data analysis methods

• Beneficial: Basic knowledge of machine learning

Literature

• Marylyn Alex, Burkhard C. Wünsche, and Danielle Lottridge. 2021. Virtual Reality Art-Making for Stroke Rehabilitation: Field study and technology probe. International Journal of Human-Computer Studies, 145, 102481. https://doi.org/10.1016/j.ijhcs.2020.102481.
• Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ‘17). Association for Computing Machinery, New York, NY, USA, 5643–5654. https://doi.org/10.1145/3025453.3025474
• Daniel Keefe, Robert Zeleznik, and David Laidlaw. 2007. Drawing on Air: Input Techniques for Controlled 3D Line Illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5 (2007), 1067–1081. https://doi.org/10.1109/tvcg.2007.1060.
• Samantha Monty, Florian Kern, Marc Erich Latoschik, Analysis of Immersive Mid-Air Sketching Behavior, Sketch Quality, and User Experience in Design Ideation Tasks, In 23rd IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE Computer Society, 2024.
• Alfred Oti and Nathan Crilly. 2021. Immersive 3D Sketching Tools: Implications for Visual Thinking and Communication. Computers & Graphics, 94, 111-123.
• Eva Wiese, Johann Habakuk Israel, Andrea Meyer, and Sara Bongartz. 2010. Investigating the Learnability of Immersive Free-Hand Sketching. In Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium (SBIM ‘10). Eurographics Association, Goslar, DEU, 135–142.
• Eun Kyoung Yang, and Jee Hyun Lee. 2020. Cognitive Impact of Virtual Reality Sketching on Designers’ Concept Generation. Digital Creativity, 31(2):82–97. https://doi.org/10.1080/14626268.2020.1726964.

Contact Persons at the University Würzburg