Developing and Evaluating an Eyes-Free Text Entry System For VR

Assigned Human-Computer Interaction Bachelor Thesis No Affiliation

Motivation/Goals

Text entry is a fundamental component of human-computer interaction, facilitating user control and information exchange with digital applications. In Virtual Reality (VR), efficient and intuitive text input techniques are particularly crucial, as immersive technologies aim to simulate real-life scenarios while maintaining the user’s sense of presence within the virtual environment. As VR applications expand beyond entertainment and gaming into areas such as work, collaboration (Horizon Workrooms, n.d.), and education (Zhou et al., 2024), there is a growing need for novel, more tailored input methods that suffice for specialized applications.
The standard for text entry in most VR applications remains a floating virtual keyboard representation, which characters are selected by directing a ray emitted from the VR headset’s controllers (Speicher et al., 2018). Although this approach has demonstrated superior performance compared to other virtual keyboard methods (Dube & Arif, 2019), it occupies a significant portion of the user’s field of view, potentially disrupting immersion. Despite efforts to reduce this issue by minimizing the keyboard (Min, 2011; Kuťák et al., 2024), a lack of text entry techniques requiring no visual attention or representation at all is given in this body of literature.
Ahmed and Farrok (2023) introduced a novel text entry method designed for visually impaired users, termed “SwingBoard.” This approach employs a radial layout with sectors representing different character sets and modes, such as numbers and letters, which are selected based on the length and angle of the user’s swipe on a smartphone. Although radial text entry interfaces are relatively uncommon in VR (Nguyen et al., 2020; Kuťák et al., 2024), they have demonstrated notable performance and user preference outcomes.
The SwingBoard approach holds great potential to be an efficient and user-friendly eyes-off text entry technique, operable with just one controller and without requiring visual attention. This capability is particularly advantageous in scenarios where users must maintain focus on specific virtual elements next to the text input field. For example, trainees in surgical simulations could make notes during training, thereby further enhancing cognitive load and thus training. Similarly, flight simulators could benefit from allowing users to input text for communication or documentation while flying. Additionally, new complex VR games can be developed where players can input commands while keeping eye on in game events and more use cases will come up following realization of this new approach.

Recent Work

Dube and Arif (2019) elaborated different categories of text entry methods, noting that physical keyboards demonstrated the highest performance in terms of words per minute. However, most VR applications are designed for operation while standing and using controllers, as is common with most consumer-grade Head-Mounted Displays (HMDs), making the use of an attached physical keyboard impractical.
In addition to the addressed issue of visual cluster, the next category, virtually displayed keyboards require high precision to accurately select keys, necessitating the user to hold the controller at a specific angle and height. This can result in physical fatigue and contribute to the “gorilla arm” syndrome, leading to discomfort and potential health issues (Eduardo et al., 2023).
Although freehand text entry - mimicking typing on a physical keyboard in mid-air - eliminates the issues of visual clutter and physical exhaustion, it has faced challenges such as high error rates due to insufficient sensor accuracy (Speicher et al., 2018). While these sensors are becoming more precise (Dudley et al., 2023), the standard for most VR applications still relies on controllers. Continuously laying down and picking up controllers can be inconvenient, particularly for users who are standing which is the norm.
Another method of eyes-free text entry that allows users to keep controllers in hand is text-to-speech. However, this technique has limitations, including the inability to enter text privately, as it requires vocalization, and a high error rate due to low word recognition accuracy and difficult error correction (Speicher et al., 2018). While integration of Artificial Intelligence (AI), such as OpenAI’s Whisper, have improved accuracy (Introducing Whisper, 2022), the broader adoption of such techniques remains questionable due to privacy concerns.

Planned Methodology

In the context of this thesis, a virtual keyboard inspired by the navigation principles of the SwingBoard keyboard will be implemented within a Unity Engine VR application (version 2022.3.46f1), using the Meta Quest 2 virtual reality headset and the Meta XR SDK as the development framework.
Unlike its original design for visually impaired users, the keyboard will be adapted for sighted users, with characters arranged in an alphabetical radial order instead of the original T9 keyboard configuration. It will be usable with one hand and the functionality of Swingboard will be adapted to the VR controllers buttons and joystick, as it will not rely on touch screen input. Moving the joystick in a specified direction a designated number of times will position the cursor on the corresponding letter. Functions such as Confirmation, Backspace, Delete, and other key operations will be mapped to the available buttons on the Quest controllers. The keyboard will be provided as a package to be easily integrated into other Unity projects.
Subsequently, an empirical user study will be conducted to evaluate the text entry performance of this keyboard, focusing on metrics such as words per minute (WPM) and error rates. Additionally, user experience, comfort, and physical effort will be assessed through questionnaires administered after the experimental sessions. The procedure will involve text entry tasks, comparing the newly developed keyboard with Meta’s standard QWERTY virtual keyboard (operated via ray interaction) and a direct touch method using hand tracking. The study will be conducted with participants recruited from the university of Würzburg. The research question could be formulated as: “Does a virtual keyboard based on the SwingBoard approach improve text entry in VR compared to Meta’s QWERTY virtual keyboard using ray interaction and direct touch?”.
This thesis aims to deliver a one-handed, eyes-free text entry solution using VR controllers, grounded in the SwingBoard approach, as an integrable Unity asset. The comparative study of text entry performance, immersion, and user preference will provide a foundation for the development of future eyes-free input methods in virtual environments. The results of the study will be presented at the HCI exposition in February 2025.

Thesis Time Schedule

• Implementation: 14.10.24 - 11.11.24 (Week 1 - 4)
• Design User Study: 28.10.24 - 10.11.24 (Week 3 + 4)
• Pilot Study: 04.11.24 - 11.11.24 (Week 4)
• Conduct User Study: 11.11.24 - 08.12.24 (Week 5 - 8)
• Evaluation of User Study: 09.12.24 - 15.12.24 (Week 9)
• Literature Research: 14.10.24 - 05.01.25 (Week 1 - 12)
• Write Thesis: 14.10.24 - 05.01.25 (Week 1 - 12)

References

• Horizon workrooms virtual office and meetings. Meta for Work. (n.d.). https://forwork.meta.com/horizon-workrooms/
• Zhou, X., Jia, R., Li, X., Tang, M., & Qi, L. (2024). Visual analysis of the evolution of application of Virtual Reality Technology in education: A bibliometric study from 2010 to 2023. Proceedings of the 2024 9th International Conference on Distance Education and Learning, 37, 257–263. https://doi.org/10.1145/3675812.3675831
• Speicher, M., Feit, A. M., Ziegler, P., & Krüger, A. (2018). Selection-based text entry in virtual reality. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3173574.3174221
• Dube, T. J., & Arif, A. S. (2019). Text Entry in Virtual Reality: A Comprehensive Review of the literature. Lecture Notes in Computer Science, 419–437. https://doi.org/10.1007/978-3-030-22643-5_33
• Min, K. (2011). Text input tool for immersive VR based on 3 × 3 screen cells. Lecture Notes in Computer Science, 778–786. https://doi.org/10.1007/978-3-642-24082-9_94
• Kuťák, D., Langlois, D., Rozič, R., Byška, J., Miao, H., Kriglstein, S., & Kozlíková, B. (2024). Design and evaluation of alphabetic and numeric input methods for virtual reality. Computers & Graphics, 122, 103955. https://doi.org/10.1016/j.cag.2024.103955
• Ahmed, I., & Farrok, O. (2023). Swingboard: Introducing swipe based virtual keyboard for visually impaired and blind users. Disability and Rehabilitation: Assistive Technology, 19(4), 1482–1493. https://doi.org/10.1080/17483107.2023.2199793
• Nguyen, A., Bittman, S., & Zank, M. (2020). Text Input Methods in Virtual Reality using Radial Layouts. 26th ACM Symposium on Virtual Reality Software and Technology. https://doi.org/10.1145/3385956.3422114
• Eduardo, Campos, A., Moraes, Í. A., Alexandre, & Marcelo. (2023). Quantifying the “Gorilla Arm” Effect in a Virtual Reality Text Entry Task via Ray-Casting: A Preliminary Single-Subject Study. https://doi.org/10.1145/3625008.3625046
• Dudley, J. J., Zheng, J., Gupta, A., Benko, H., Longest, M., Wang, R., & Kristensson, P. O. (2023). Evaluating the performance of hand-based probabilistic text input methods on a mid-air virtual QWERTY keyboard. IEEE Transactions on Visualization and Computer Graphics, 29(11), 4567–4577. https://doi.org/10.1109/tvcg.2023.3320238
• Introducing whisper. OpenAI. (2022, September 21). https://openai.com/index/whisper

Contact Persons at the University Würzburg