Augmented Reality Agents in Medicine

Assigned Human-Computer Interaction Master Project Thesis No Affiliation

Background

Augmented and virtual reality can support physicians during their daily work. Especially augmented reality can enhance existing workflows, e.g., during operations (1). During COVID-19, augmented reality applications were tested during ward rounds, which led to lower infection rates, faster ward rounds, and a better experience for medical students (2). These applications could be further enhanced by embodied KI, i.e., agents, which are more natural to interact with and perceived as more trustworthy (3). This could be used to increase the acceptance of AI among medical professionals.

This project or thesis aims to develop an augmented reality prototype and evaluate it with students and medical personnel. The user should work with a virtual agent in one of two scenarios: The first scenario is a continuation of the work done in the DZ.PTM project, where a mobile app was developed to support doctors during their ward rounds (4). A virtual assistant that helps doctors request examinations and document wounds should be developed during the thesis or project. Another possible project or thesis could be to investigate how virtual agents affect teams. For this, a virtual agent needs to be developed as a team member in the so-called “team timeout” - a short, structured team meeting right before an operation to clarify open questions and ensure the correct organs are operated on (5). This project will take place within the research projects DZ.PTM and AIL@WORK and is done in cooperation with the University Hospital Würzburg.

Tasks

This project will focus on the following tasks:

• Literature research about the theoretical background
• Development of a mixed reality prototype
• Execution of a explorative user study in the university hospital
• Evaluation and presentation of results

Prerequisites

• Experience with the Unity game engine
• Experience with conducting user studies
• Optional: Experience with the REST API
• Optional: Experience with developing AR applications

Literature

• 1: Oren M. Tepper, Hayeem L. Rudy, Aaron Lefkowitz, Katie A. Weimer, Shelby M. Marks, Carrie S. Stern, and Evan S. Garfein. “Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room.” Plastic and reconstructive surgery 140, no. 5 (2017). 1066-1070.
• 2: Jeremy B. Levy, Edmund Kong, Nathan Johnson, Ashni Khetarpal, James Tomlinson, Guy FK Martin, and Anisha Tanna. The mixed reality medical ward round with the MS HoloLens 2: innovation in reducing COVID-19 transmission and PPE usage. Future healthcare journal 8, no. 1 (2021). e127.
• 3: Ella Glikson, and Anita Williams Woolley. Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals 14, no. 2 (2020). 627-660.
• 4: Philipp Krop, Samantha Straka, Melanie Ullrich, Maximilian Ertl, Marc Erich Latoschik, IT-Supported Request Management for Clinical Radiology: Contextual Design and Remote Prototype Testing, In CHI Conference on Human Factors in Computing Systems Extended Abstracts (45), pp. 1-8. New York, NY, USA: Association for Computing Machinery, 2022.
• 5: WHO Surgical Safety Checklist and Implementation Manual. http://www.who.int/patientsafety/safesurgery/ss_checklist/en 

Contact Persons at the University Würzburg