Evaluating the effect of AR-assistance on task performance in manual liquid handling scenarios: a comparative study

Assigned Human-Computer Interaction Master Thesis No Affiliation

Outline

• 1. Motivation
• 2. Related Work
• 3. Preliminary Project Work
• 4. Methodology
• 5. Conclusion
• References

1. Motivation

Research laboratories in the Life Science industry discover new medicine through accurate, reliable, and reproducible science. Manual liquid handling , despite growing automation efforts, is still a key and critical process in this endeavour. Lab technicians need high concentration, good preparation, and years of practice to avoid mistakes [1]. Despite all preparations, errors in the liquid handling process do occur. Unfortunately, those mistakes are often not discovered until the end of the analytical pipeline, rendering the results unreliable at best and useless at worst. Insufficient data can lead to an expensive and time-consuming rerun of the experiment, keeping the researchers from doing more progressive work [2]. While lab automation with robots is often considered a very accurate and reliable solution to this problem, the upfront investment, high maintenance cost and inflexible programming cut out small sized research laboratories with frequently changing experiments [3]. But, even in small sized laboratories there are ‘bio-robots‘ that are capable and highly flexible in learning and executing a wide range of tasks. Those ‘bio-robots’ - us humans - often struggle with complex tasks that require a high level of focus over a long period of time. Loss of concentration is responsible for many mistakes we do every day. To solve this, humans come up with a variety of cognitive artifacts as support in their work: post-it’s, paper notes or specific placement of objects etc. [4]. In the laboratory context the most common form of cognitive artifact is the laboratory notebook (LN, see figure below). It’s purpose is to keep a permanent and detailed record of the materials, procedures, results obtained and any observations made by a scientist during an experiment [5].

The laboratory notebook is an important tool for the scientist in order to execute experiments accurately. For a long time only existent in paper (pLN), nowadays the industry slowly but steadily shifts towards accepting electronic lab notebooks (eLN) to reap their benefits [6]. Although eLNs have many positive features, one major drawback they share with their paper counterparts is the spatial disconnection between where the information is stored and where it is needed. In the case of pipetting, scientists have to refocus from their sample they are working on, towards the notebook in order to make sure they are following the documented protocol. In recent years, researchers have explored possibilities to extend the functionality of eLNS with wearable devices such as Google Glass [7] or the Apple Watch [8]. The goal was to provide important information to the lab technician with as little friction and spatial separation as possible. Besides getting protocol information in the field of view or on the lab technicians wrist, a combination of both wearable devices was used to extend input modalities and control experiment documentation [9]. Another way of bridging the gap between information storage and application is the use of Augmented Reality (AR). Motivated by the research from other scientist on the benefits of AR in the laboratory environment, our previous work looked at ways how AR on an optical see-through head mounted display (OST HMD) could assist scientists to perform manual liquid handling task faster, with less errors and less cognitive load compared to the use of paper based experiment protocols [10]. Building upon the previous system, this work will further explore and evaluate the benefits of AR-based assistance in manual liquid handling scenarios. We will conduct a comprehensive user study, comparing our AR solution with a state-of-the-art, screen-based pipetting assistance system and a traditional paper-based protocol. By doing so, we hope to contribute to the growing body of evidence supporting AR as a valuable tool in modern laboratory practice.

2. Related Work

The laboratory environment and its potential for digital development have been the topic of many research works. Although we want to focus on manual liquid handling within the laboratory workflow in this work, we have to contextualize of what this process step is a part of. Pipetting is one of many steps that make up a lab experiment: from planning, to sample and reagent preparation, execution, documentation and analysis. All these steps are equally important and share single point of contact: the laboratory notebook. Studies have discussed ways of replacing established paper lab notebooks (pLN) with electronic ones (eLN) to harness their transformational potential [11][12]. Efforts to digitalize the laboratory notebook and its use are motivated by a holistic approach to modernize laboratory work and lay the groundwork for other processes to follow. Similarly, Smart tabletop surfaces have been developed and evaluated as ways to digitize and augment the experiment workflow and to assist in pipetting workflows [13][14]. Smart work benches were imagined to enable new forms of collaboration, a connection point for a variety of lab data to flow through and a source of assistance for lab technicians, not only but importantly for manual pipetting. It was shown that manual pipetting suffers from inter- and intraindividual imprecision [15], especially when using mechanical pipettes. Lately, electronic pipettes have become commercially available with some even having wireless connectivity to receive and transmit information [16][17]. While these devices minimise volumetric inconsistencies, they neither shield the lab technicians from the substantial mental load exerted by long periods of pipetting nor prevent them from misplacing liquids on the sample carrier [1]. Commercially available products such as the Gilson Trackman with connected electronic pipettes, as well as research projects such as the Pipette Show, aim to assist humans in the liquid handling workflow by placing the sample carrier on a screen [18][19]. While these solutions can improve experiment accuracy and may relieve some of the mental load, similar to tabletop solutions, they are limited by their screen size, somewhat immobile nature, and single purpose. Despite its proven potential in various applications, such as surgical settings [20], the adoption of AR in liquid handling has been relatively limited. One notable example is a projector-based prototype that utilized markers to track the sample carrier and pipette tip, facilitating interaction and guidance in AR [21]. Although the initial testing and user feedback were positive, the study called for further improvements and exploration. The advent of wearable AR devices, such as the HoloLens 2, coupled with the scarcity of research on AR-assisted manual liquid handling, served as the primary motivation for our previous work, which is detailed in the following section.

3. Preliminary Project Work

In earlier work, we set out to investigate the research question what effect augmented reality assistance has on task performance in manual liquid handling scenarios [10]. Task performance was comprised of and defined as:

(1) Task execution time: the time needed by each participant to complete the test experiment measured in minutes and seconds (2) Task execution accuracy: measured by how many misplacement errors were made by each participant within the test experiment (e.g. missing wells or skipping steps entirely).

Choosing these metrics was motivated by their inherent value in achieving accurate, reliable and reproducible science. If less time is spent handling liquids it can be spent more on analyzing results and progressing the research. But most importantly, if errors in the liquid handling stage can be minimized or even eradicated, no reruns are needed saving significant amounts of time and making the analysis more robust as the data is not skewed by uncaught pipetting mistakes. We proposed a solution to assist in the liquid handling process and setup a small evaluation experiment with expert users from the Biocenter at the Julius-Maximilians University Würzburg [22]. Using the Microsoft HoloLens 2 [23] as HMD, lab technicians were shown instructive augmentations on top of their sample carrier (96-well Micro Tither Plate; MTP) while proceeding to pipette a short, theoretical experiment protocol (external view and first-person view in the left figure below).

The evaluation experiment used a theoretical pipetting protocol consisting of 30 liquid transfer steps, using a single channel electronic pipette and a single, pre-filled 96-well MTP. 9 participants were tested in a within-subject design, in randomized starting order, pipetting two equally complex variations of the 30 step protocol in both the Paper-method (instructions printed on a A4 paper sheet, see right figure above) and AR-method (instructions augmented on top of the sample carrier, see left figure above)

Previous research questions

The study was designed to answer the following research question through quantitative analysis and further the understanding of the challenges faced within laboratory environments through semi-structured qualitative interviews with each participant.

(1) Is AR-assisted pipetting faster compared to paper-based pipetting? (2) Is AR-assisted pipetting more accurate (less misplacement errors) compared to paper-based pipetting? (3) Is AR-assisted pipetting less cognitively taxing compared to paper-based pipetting?

Previous findings

Compared to working with a paper-based protocol, the evaluation showed promising results. The data suggested, that AR-assistance can reduce task completion time and potentially eradicate misplacement errors. The sample size was too small to statistically proof any of the tendencies shown through descriptive analysis of the data. Therefore the key findings of the study can only be formulated as suggestions.

(1) The experiment data suggest that participants are faster using the AR-assisted method (left figure below). (2) The experiment data suggest that participants make less misplacement errors using the AR-assisted method, hence are more accurate (right figure below). (3) No significant difference in experienced cognitive load could be inferred from the gathered NASA TLX questionnaire data.

Previous limitations

Although the experiment was leaning in favor of the posed research question, the study highlighted the following key limitations to be improved in future work in order to solidify the suggested tendencies:

(1) Sample size and design: only 9 participants were tested in a within-subject design. A larger sample size and independent groups were recommended to make potential findings more robust. (2) Protocol duration: the participants spent less than 10 minutes pipetting. In order to observe the long term effects of assistance on concentration degradation and loss of focus, the pipetting duration should be increased significantly (increase number of steps and/or complexity). (3) Real world alignment: the liquid transfers in the experiment were laid out artificially and did not resemble a structured experiment typically conducted in a laboratory. The author suggested to design the protocol more in line with an actual experiment found within the laboratory work context. (4) User Experience: as the proposed system was a first iteration, the author suggested improvements to be made regarding visualization and user feedback in order to enhance the user experience.

The goal of this study is to build upon the key findings of the previous work while addressing some of the highlighted limitations, specifically sample size, protocol length and real world alignment. Additionally, we introduce a new comparative aspect to the research question, that is, what are the effects of using different types of assistive technology on task performance in manual liquid handling scenarios?

4. Methodology

The general working plan is laid out in the figure above. In the following, the preliminary plan of action is briefly described in each section. First, we will design a more extensive pipetting protocol, change the Unity application accordingly and setup a self-hosted version of the Pipette Show project for our experiment. Similar to our previous work, we conduct an experiment to evaluate the effect on pipetting task performance depending on the used method/level of assistance. Three methods will be compared against each other: Paper, Screen and AR based assistance. Participants are sourced from the biochemical research field and pre-selected by their pipetting experience and regular exposure to pipetting. The gathered data will be analyzed according to the posed research questions and results will be put into context of previous and related work as well as the current state of digital Life Science laboratories.

Protocol design and system implementation

Our previous protocol entailed 30 single liquid transfers within one 96-well MTP. Participants executed this task within 4 - 9 minutes. Although positive effects of AR assistance were suggested from the data gathered, we hope to make the effect more clear through a prolonged experiment. To achieve this, an experiment protocol with an estimated execution time between 20 to 30 minutes is developed. To make the results of this study accessible for interpretation by researchers in the biochemical field, we develop the new protocol in cooperation with lab technicians to ensure a closer alignment to the real-world context. The complexity of the protocol needs to be equal no matter of the method of execution being used.

Setup and test Study

Screen-based assistance for comparison

We introduce a comparative aspect to this study to assess the performance of already available pipetting assistance solutions. Products, such as the Pipette Show or Gilson Trackman are already available for labs to experiment with. Both products fall into the category of screen-based pipetting assistance. The relevant pipetting protocol information is shown on screen in a layout that fits the pyhsical dimensions of a 96-well MTP. Colored visualization shown on screen can be observed by the user through the transparent well bottom of the sample carrier. Besides augmenting the sample carrier from below, users are presented with additional protocol information like volume, sample name, alphanumeric positions etc. For this study we use the Pipette Show project as basis for the screen-based condition. As it is Open-Source and includes a separate protocol builder application, it provides the needed functionality and flexibility for our experiment. In total, three conditions (levels of assistance) are examined in this experiment. Paper, Screen and AR. By including the Screen condition, we hope to make the interpretation of any potential effects of our proposed AR system more accurate. Comparing Screen and AR could potentially yield interesting results, as screen-based pipetting assistance may just be as effective as the AR solution. Further, we assess the potential differences in product usability and desirability using AttrakDiff [24] as well as the experienced cognitive load using the NASA TLX questionnaire [25][26].

Sample size and experiment design

A larger pool of participants is critical in making the results more robust compared to previous work by Hile et al. and ourselves. Due to the special nature of the domain this study is situated in, finding a large enough group of participants, which need to be trained and experienced professionals, is difficult. A Between-Subject design is preferred to minimise potential learning effects and fatigue of the individual participants, as the protocol length increases significantly. Therefore we end up with three individual participant groups, one for each condition (Paper, Screen and AR). Ideally, we aim to recruit up to 30 participants per group, 90 participants in total. This would allow for a sound statistical analysis and well interpretable results. Potential participants will be screened and selected by a questionnaire to find a relatively homogeneous group of people who have sophisticated and regular pipetting experience (~1-3 pipetting sessions per week, ~30+ min/session). As of now, there are two possible ways for recruitment. The first would be to recruit individual candidates from the Bio-Center at JMU. We would mostly expect PhD students to be among the responders. The second option is a collaboration with Boehringer-Ingelheim at the Biberach research facility [27]. As this location is a trainee site, it may be possible to source lab technician trainees in their last year of training as well as seasoned professional from wet labs on the location. This would provide a quite homogeneous pool of well trained professionals for the experiment lending the results even more significance. Talks with potential partners are ongoing.

Conduct Study

As of now, the rough time frame for conducting the study is around July and August. The specific time of execution is currently bound by the talks with Boehringer-Ingelheim and the progress of protocol and AR system development. Regardless of other factors, the time resources for conducting the experiment and data gathering are estimated as follows: 15 min onboarding, 30 to 45 min pipetting, 15 to 30 min debriefing per participant, summing up to 60 to 90 min per session, 90 to 135 hours of experiment work in total. These estimates may change in accordance to limitations in either recruitment of participants and or changes in experiment design.

Data analysis and thesis writing

Given the motivation, previous and related work as well as using the outlined methodology above, this work tries to answer the following research questions:

(1) Is there a difference in task performance dependent on the assistance level along either dimension of: (a) Time (measured in minutes:seconds) (b) Accuracy (measured in misplacement errors) (2) Is there a difference in experienced mental load dependant of the assistance level? (3) Is there a difference in the usability rating between at least one of the techniques?

After the experiment is conducted successfully, the data gathered will be analyzed in accordance with the stated research questions. The results will be critically reviewed and placed in context of the overall research done in this field. We expect much of the contextual and descriptive part of this work to be written before or while the experiment is conducted, with the analysis and interpretation part following in September and October.

5. Conclusion

Motivated to build upon previous work and the apparent potential to transform one of the key processes in laboratory experiments, this study aims to investigate the effect of varying levels of assistance on pipetting task performance, focusing on three different conditions: Paper, Screen, and AR based assistance. By designing a comprehensive experiment protocol in collaboration with lab technicians and recruiting a large pool of domain specific participants with significant pipetting experience, we hope to obtain robust and interpretable results. The study will compare task performance (time and accuracy), mental load, and usability ratings among the three conditions to determine which method is most effective. The time frame for developing the software system and experiment design is estimated to be around May and June, conducting the study in July and August, with data analysis and thesis writing following in September and October. The outcomes of this research will not only contribute to a better understanding of the potential benefits of AR technology in life science laboratory environments but also provide valuable insights into the effectiveness of various assistance methods in improving task performance and user experience. By comparing the results to existing screen-based assistance solutions, we hope to shed light on whether AR technology offers significant advantages over current methodologies, thereby potentially informing future development and implementation of AR-based laboratory assistance tools.

References

[1] Manida Swangnetr, David Kaber, Ellen Vorberg, and Heidi Fleischer, and Kerstin Thurow. “Workload Assessment for Manual and Automated Processes in Life Sciences.” In: Advances in Physical Ergonomics and Human Factors: Part I (2018), p. 121.

[2] Houriiyah Tegally, James Emmanuel San, Jennifer Giandhari, and Tulio de Oliveira. “Unlocking the efficiency of genomics laboratories with robotic liquid-handling.” In: BMC genomics 21 (2020), pp. 1–15.

[3] Ian Holland and Jamie A Davies. “Automation in the life science research laboratory.” In: Frontiers in Bioengineering and Biotechnology 8 (2020), p. 571777.

[4] Donald A Norman. “Cognitive artifacts.” In: Designing interaction: Psychology at the human-computer interface 1.1 (1991), pp. 17–38.

[5] Colin L Bird, Cerys Willoughby, and Jeremy G Frey. “Laboratory notebooks in the digital era: the role of ELNs in record keeping for chemistry and other sciences.” In: Chemical Society Reviews 42.20 (2013), pp. 8157–8175.

[6] Michael Rubacha, Anil K Rattan, and Stephen C Hosselet. “A review of electronic laboratory notebooks available in the market today.” In: JALA: Journal of the Association for Laboratory Automation 16.1 (2011), pp. 90–98.

[7] Grace Hu, Lily Chen, Johanna Okerlund, and Orit Shaer. “Exploring the use of google glass in wet laboratories.” In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 2015, pp. 2103–2108.

[8] Santiago Guerrero, Gwendal Dujardin, Alejandro Cabrera-Andrade, César Paz-y-Miño, Alberto Indacochea, Marta Inglés-Ferrándiz, Hima Priyanka Nadimpalli, Nicola Collu, Yann Dublanche, Ismael De Mingo, et al. “Analysis and implementation of an electronic laboratory notebook in a biomedical research institute.” In: PLoS One 11.8 (2016), e0160428.

[9] Philipp M Scholl, Matthias Wille, and Kristof Van Laerhoven. “Wearables in the wet lab: a laboratory system for capturing and guiding experiments.” In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2015, pp. 589–599.

[10] Sascha Lange and Florian Niebling. Augmented Reality Pipetting Assistance for Life Science Laboratory Applications. HCI Project. Julius-Maximilians University Wuerzburg, 02/2023.

[11] Samantha Kanza, Cerys Willoughby, Nicholas Gibbins, Richard Whitby, Jeremy Graham Frey, Jana Erjavec, Klemen Zupančič, Matjaz Hren, and Katarina Kovač. “Electronic lab notebooks: can they replace paper?” In: Journal of cheminformatics 9.1 (2017), pp. 1–15.

[12] Björn Gerlach, Christopher Untucht, and Alfred Stefan. “Electronic Lab Notebooks and Experimental Design Assistants.” In: Good Research Practice in Non-Clinical Pharmacology and Biomedicine. Springer International Publishing, 2019, pp. 257–275. DOI: 10.1007/164 2019 287. URL: https://doi.org/10.1007%2F164 2019 287.

[13] Aurélien Tabard, Juan-David Hincapié-Ramos, Morten Esbensen, and Jakob E. Bardram. “The eLabBench.” In: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 11/2011. DOI: 10.1145/2076354.2076391. URL: https://doi.org/10.1145% 2F2076354.2076391.

[14] Wendy E. Mackay, Guillaume Pothier, Catherine Letondal, Kaare Bøegh, and Hans Erik Sørensen. “The missing link.” In: Proceedings of the 15th annual ACM symposium on User interface software and technology. ACM, 10/2002. DOI: 10.1145/571985.571992. URL: https://doi.org/10.1145% 2F571985.571992.

[15] Giuseppe Lippi, Gabriel Lima-Oliveira, Giorgio Brocco, Antonella Bassi, and Gian Luca Salvagno. “Estimating the intra- and inter-individual imprecision of manual pipetting.” In: Clinical Chemistry and Laboratory Medicine (CCLM) 55.7 (01/2017). DOI: 10 . 1515 / cclm - 2016 - 0810. URL: https://doi.org/10.1515%2Fcclm-2016-0810.

[16] Picus NxT Electronic Pipette. 02/05/2023. URL: https : / /www. sartorius . com/ en / products / pipetting / electronic - pipettes/picus-nxt (visited on 02/05/2023).

[17] PIPETMAN M P100M BT Connected. 02/05/2023. URL: https : / / de . gilson .com/DEDE/ pipetman -m- p100m - bt - connected.html (visited on 02/05/2023). < a id=”ref18”></a> [18] Johannes Falk, Marc Mendler, and Johannes Kabisch. “Pipette Show: An Open Source Web Application to Support Pipetting into Microplates.” In: ACS Synthetic Biology 11.2 (2022), pp. 996–999.

[19] TRACKMAN Connected. 02/05/2023. URL: https : / / de . gilson . com / trackman - connected . html (visited on 02/05/2023).

[20] P. Vávra, J. Roman, P. Zonča, P. Ihnát, M. Nemec, J. Kumar, N. Habib, and A. El-Gendi. “Recent Development of Augmented Reality in Surgery: A Review.” In: Journal of Healthcare Engineering 2017 (2017), pp. 1–9. DOI: 10.1155/2017/4574172. URL: https://doi.org/10.1155% 2F2017%2F4574172.

[21] Harlan Hile, Jiwon Kim, and Gaetano Borriello. “Microbiology Tray and Pipette Tracking as a Proactive Tangible User Interface.” In: Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2004, pp. 323–339. DOI: 10. 1007/978- 3- 540- 24646- 6 23. URL: https://doi.org/10. 1007%2F978-3-540-24646-6 23.

[22] Biocenter, JMU. 04/16/2023. URL: https : / / www . biozentrum.uni-wuerzburg.de/en/about-the-biocenter/thebiocenter- today/ (visited on 04/16/2023).

[23] HoloLens 2. 04/16/2023. URL: https://www.microsoft.com/ de- de/hololens/hardware (visited on 04/16/2023).

[24] Ignacio Diaz-Oreiro, Gustavo Lopez, Luis Quesada, and Luis A Guerrero. “Standardized questionnaires for user experience evaluation: A systematic literature review.” In: Multidisciplinary digital publishing institute proceedings 31.1 (2019), p. 14.

[25] Sandra G Hart. “NASA-task load index (NASA-TLX); 20 years later.” In: Proceedings of the human factors and ergonomics society annual meeting. Vol. 50. 9. Sage publications Sage CA: Los Angeles, CA. 2006, pp. 904–908.

[26] Sandra G Hart and Lowell E Staveland. “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.” In: Advances in psychology. Vol. 52. Elsevier, 1988, pp. 139–183.

[27] Boehringer Ingelheim. 08/01/2022. URL: https : / /www. boehringer - ingelheim . com / de / ueber - uns / standorte - weltweit/standort-biberach (visited on 04/16/2023).

Contact Persons at the University Würzburg