Reproduce, Optimize, Expand: Revisiting a multi-session behavioral biometric dataset for user identification in VR

Assigned Human-Computer Interaction Computer Science Master Thesis No Affiliation

Introduction

Especially with consumer-grade hardware, Virtual Reality (VR) gained more and more relevance in recent years. As a result, content verity and use cases for VR are increasing as well. Many of those could benefit from or even require knowledge of the users’ identity (for example e-commerce, therapy, rehabilitation, social VR, and gaming). Especially considering VR setups as workspaces shared by multiple users, the importance of user identification becomes apparent.

Identification processes in general can be distinguished in three different categories. Most common is knowledge based identification like passwords, pins or patterns. The second category uses biometrical features of a user, such as fingerprints or iris details. A third category relies on behavioral biometrics. Here, unique patterns within the specific interaction of a user are leveraged for identification. The first category is the simplest in terms of implementation, but has some drawbacks, especially for VR applications. Entering a password or pin in VR requires some sort of text input method suited for VR. Even though many different approaches have been investigated^{1 2 3 4}, the speed and comfort of typing in a desktop like environment are still unmatched by VR methods. The main drawback of pattern based processes (e.g. drawing a pattern via conecting dots in a 3x3 grid) rises security concerns, since the user in VR is not aware of the surrounding real world and potential bystanders⁵. The use of biometrical features is almost non intrusive and fast, but requires additional hardware like fingerprint readers, which are not typically included in VR setups. Behavioral Biometrics share the benefits of static biometrical approaches, but do not require additional hardware. The principle behind behavioral biometrics is that the details in behavior while pursuing a specific task are unique to individuals. In terms of virtual reality, such details are movement patterns in 3D space.

Related work

Previous work in this field has already shown that the 3D-spatial information about a users’ interaction naturally recorded by any VR setup is sufficient for user identification in many scenarios (see table 1).

Table 1: Overview of related work. SR, BR and BRV indicate feature computations (scene-relative, body-relative, body-relative velocity), N indicates the number of individual participants.
Origin: Schell et al. (2021)⁶

However, following the discussion of Schell et al.⁶ about previous work, there are several shortcomings which should be addressed: Firstly (1), most results so far were not validated against data recorded on a separated session. Although discussed by some authors^{6 7 8 9}, the impact of different pre-processing approaches on multi-session scenarios needs further investigation. Second (2), only Schell et al.⁶ employed proper hyperparameter optimization for their machine learning classifier. Yet this process is key for unveiling the true potential of any machine learning approach. Third (3), only Schell et al.⁶ and Liebers et al.⁸ used publicly available datasets. Therefore, none of the other previous work can be validated through reproduction work. This is especially problematic, since machine learning based solutions are very sensitive for flaws in terms of development and methodology.

To address those shortcomings, I will focus on the work of Liebers et al.⁸ and Schell et al.⁶ With the dataset of Liebers et al. being publicly available, I can investigate the reliability of the results, especially considering the multi-session nature of their dataset. Then, with the enhanced machine learning methodology described by Schell et al., I can investigate the validity and optimization potential of the results of Liebers et al. In a third stage, I can use the original dataset to explore and compare against different pre-processing approaches, suggested by related work.

Liebers et al. (2021)

Liebers et al.⁸ investigated the impact of users’ physiology on behavioral biometric user identification systems. Since my work will be based upon the dataset created by them, I will now briefly describe it. Two simple VR games, archery and bowling, were developed (scenarios). Shooting of a single arrow, or rolling the ball once, were considered a single repetition for the corresponding scenario. As the authors investigated in effects of physiology, they created 4 conditions by manipulating the virtual avatars of the users:

1. HeightNormalization (HN), in which the height of the avatar was artificial set the be same for every participant,
2. ArmLengthNormalization (AN), in which the acceleration of the virtual hands was normalization across all participants, effectively granting each participant the same arm span,
3. BothNormalizations (BN), a combination of 1. and 2.
4. WithoutNormalization (WN), in which no avatar manipulation took place.

For both scenarios, each of the 16 participants did 12 repetitions in each condition. The same recording was then repeated on a second day. The data consists of the positional and rotational data of the hand held controllers and the head mounted display, sampled at 72hz during the individual repetitions. Along with this motion data, timestamps and a discrete phase variable (e.g. (a) pick up the arrow, (b) tension the bow, (c) aim, (d) shoot) were added to every sample. For training purposes, 4 combinations of motion data, timestamp and phase were separated as individual feature sets (see Table 3).

Research Questions

1. How reliable are the results of Liebers et al.⁸?
2. a) To which extent can the results of Liebers et al.⁸ be improved via adequate hyperparameter optimization?
   b) Dose the HeightNormalization condition stay superior to the WithoutNormalization condition in terms of identification accuracy, if hyperparameter optimization was employed for both?
3. How do velocity and acceleration approaches compare on the dataset of Liebers et al.⁸?

Methodology

All research questions will be perused individually.

Question 1 - Replication of Liebers et at (2021)

In order to gain insight into the reliability of the original results, a replication attempt will be made. Originally, Liebers et al. investigated in both multi-layer perceptron (MLP) and recurrent neural networks (RNN) architectures. For my replication, I will focus on RNN, as they were found to perform significantly better in the original work⁸. This behavior is expected to hold true for sequential data in general¹⁰, and was found for an similar task by Schell et al.⁶.

Table 2 shows the accuracies for the various combinations of condition and scenario achieved by Liebers et al. The individual cells in the blue area summarize the configurations targeted by Research Question 1 (MLP columns are not considered).

Question 2 - Optimization on Liebers et al. (2021)

Liebers et al⁸ do not report on hyperparameter optimization. Since this process usually reveals the true potential of machine learning, I will search for optimized parameters. This will lead to a new set of trained models. To perform this optimization, the available data must be split into three data sets. All recordings from the second session will be kept as a test dataset, reserved for comparisons of the final results. The data from the first session will be split into a training and a validation dataset for training and optimization purposes.

To keep the number of unique combinations within the scope of this work, I will not optimize the parameters for each condition introduced by Liebers et al. Instead, I will focus on the WN condition. However, the specific trials Archery+F3+HeightNormalization and Bowling+F2+HeightNormalization will also be considered. According to the authors⁸, they achieved the best accuracies for the archery and bowling tasks respectively. By optimizing for these conditions, a comparison can be made with WN trials. The individual results of Liebers et al. are summarized in Table 2 (highlighted in orange).

The hyperparameter optimization process is divided into two stages following the work of Schell et al⁶. The first stage deals with the parameters of the architecture of each model, while the second stage searches for data-related parameters. The individual parameters and search spaces are listed in Table 4. In the first stage, all data parameters are frozen in the state they were used by Liebers et al., while architectural parameters are searched for each scenario (archery/bowling) and feature set (see Table 3) combination of the WN data, as well as for the two HN configurations mentioned above. In the second phase, the best architectural parameters found for each combination are used, and data parameters for each combination are searched.

Question 3 - Comparing positional data against velocities and acceleration

For the final Research Question, I will derive velocity and acceleration data based on the WN motion data. The 4 feature sets will stay the same, except for the positions and angles, which will be exchanged with the velocities in one condition and with acceleration data in an other. In doing so, I follow suggestions made by other work so far^{11 9 7 6}. The same optimization procedure as mentioned in Question 2 will be employed here.

This will produce 16 new classifiers, which are illustrated by the green additions in table 2. In a final comparison, all of the optimized results will then be compared and discussed.

Literature

Contact Persons at the University Würzburg