Original Research
Frontiers in Virtual Reality ·
Volume 7 - 2026 | https://doi.org/10.3389/frvir.2026.1918264
A real-time error prevention system for gaze-based interaction in virtual reality based on anomaly detection
- Björn R. Severitt1*corresponding author
- Yannick Sauer2
- Nora Castner2
- Siegfried Wahl1,2
- 1ZEISS Vision Science Lab, University of Tübingen, Tübingen, Germany
- 2Carl Zeiss Vision International GmbH, Aalen, Germany
Abstract
Gaze-based interaction enables intuitive, potential hands-free control in immersive environments, but remains prone to unintended inputs, commonly referred to as the Midas touch problem. To address this limitation, we propose a real-time Error Prevention System (EPS) that detects anomalous gaze behaviour during selection tasks. The EPS is based on a Temporal Convolutional Network Autoencoder (TCNAE) trained to reconstruct typical gaze dynamics of correct selections. In a Virtual Reality (VR) visual search task, 37 participants used three gaze-based interaction methods - dwell time, gaze and head direction alignment, and nod - each evaluated with and without the EPS. Both objective performance metrics and subjective user evaluations were collected. The EPS reduced incorrect selections by up to 95% for dwell time and gaze and head alignment, and led to significant performance improvements in these conditions. Effects for nodding were smaller and more variable. Overall, participants rated the system positively, reporting improved confidence and usability, while system performance and benefits varied across individuals. These findings demonstrate that anomaly-based error prevention can effectively reduce incorrect gaze-based selections in real time without disrupting interaction. However, the variability across interaction methods and users highlights the need for adaptive approaches. The proposed method shows strong potential for improving gaze-based interfaces in VR, Augmented Reality (AR), and assistive technologies.
Keywords
1 Introduction
Gaze-based interaction offers an intuitive and hands-free way to communicate with digital systems, especially in immersive environments and assistive technologies where traditional input devices are impractical (Severitt et al., 2024a; Sidenmark et al., 2021). Despite its potential, gaze interaction still faces challenges in accurately interpreting the user’s intentions. A well-known issue is the Midas touch problem (Jacob, 1990), in which unintended gaze behaviour is mistakenly registered as intentional input.
To mitigate such errors, various confirmation mechanisms have been proposed, including head gestures, dwell times, and combined gaze-head alignment (Sidenmark and Gellersen, 2019; Špakov and Majaranta, 2012; Esteves et al., 2015). However, even with these mechanisms, false selections persist and can undermine user trust—especially in collaborative or safety-critical scenarios (Padovani et al., 2019; Flook et al., 2019).
Given previous findings that gaze behaviour reflects signs of confusion or correction even before a selection is confirmed (Severitt et al., 2024b), we focus our anomaly detection approach on the immediate temporal vicinity of the selection event to enable early error detection. In our previous work (Severitt et al., 2025a), we presented a theoretical proof-of-concept for using anomaly detection to identify erroneous gaze-based selections, defining errors as outliers in gaze dynamics. This concept was further substantiated by empirical results in (Severitt et al., 2025b), where we showed that even with confirmation mechanisms, selection errors frequently occurred, negatively impacting the user experience.
At the same time, the success of gaze-based interaction depends not only on technical accuracy but also on the quality of the user experience. Error prevention mechanisms may be most effective when they work in the background, reducing errors without disrupting interaction. Ideally, users can continue to focus on their task while the system silently supports them, contributing to a more natural and trustworthy experience. From this perspective, both reliable anomaly detection and unobtrusive support are important. The goal of this work is to investigate whether anomaly-based error detection can be used as a real-time error prevention mechanism in gaze-based interaction, using a Temporal Convolutional Network Autoencoder (TCNAE) as a concrete instantiation. While prior work has demonstrated the feasibility of detecting erroneous gaze-based selections offline, it remains unclear whether such approaches can be integrated into real-time interaction and how their effectiveness depends on the underlying interaction method.
On this basis, the present study evaluates a real-time Error Prevention System (EPS) based on an Auto Encoder (AE) with a Temporal Convolutional Network (TCN), a model architecture that has proven effective for anomaly detection in time series data (He and Zhao, 2019). The system is trained to reconstruct typical gaze patterns for correct selections, with deviations from this learned representation interpreted as potential errors. We implement and evaluate the EPS in a Virtual Reality (VR) visual search task and compare three gaze-based interaction methods—Dwell Time, Gaze and Head and Nod—with and without the system. In particular, we investigate whether anomaly-based error detection can be effectively integrated into real-time interaction and how its effectiveness depends on the interaction method. Specifically, we address two central questions: (1) How do participants experience and evaluate the system in terms of usefulness, trust, and overall preference, and (2) can the EPS reliably prevent errors in real-time environments across different interaction methods?
2 Related work
Gaze-based interaction has become increasingly important in various fields, including assistive technologies (Parisay et al., 2021; Istance et al., 2008; Rajanna and Hammond, 2016), Augmented Reality (AR)/Virtual Reality (VR) (Mohan et al., 2018; Velloso et al., 2016; Gardony et al., 2024; Schweigert et al., 2019), and Human-Computer Interaction (HCI) in general (Gemicioglu et al., 2023). An important goal of these approaches is to reduce unintended selections, known as the Midas touch problem, in order to improve both the user experience and trust in the system. Recent studies also show that features derived from eye-tracking data can be used to infer user intentions David-John et al. (2021); Bovo et al. (2020) and detect interaction errors (Severitt et al., 2023; Bovo et al., 2020; Bruder et al., 2016), highlighting the potential of gaze as a powerful input modality.
Recent studies have demonstrated the potential of machine learning to improve gaze-based input by contextualizing gaze behaviour within the interaction environment. For example, Shevtsova et al. (2025) shows that machine learning techniques were applied to gaze data with free behaviour in a visually complex game environment, resulting in a significant reduction in false alarms and an improvement in user satisfaction—all without explicit intention confirmation. Such results underscore the increasing suitability of data-driven methods for improving gaze-based interaction.
In addition, sequence modeling techniques, such as TCN were used to classify user errors and predict intentions directly from raw gaze histories, achieved performance well above chance levels (Sendhilnathan et al., 2022). Continuing this work, Sendhilnathan et al. (2025) demonstrated that incorporating multimodal inputs into the TCN framework further improves error classification. Although these approaches are effective, they are generally based on a fixed taxonomy of error classes, which makes it difficult to generalize them to previously unknown or unexpected interaction errors.
In contrast, our approach reframes selection errors not as predefined classes, but as anomalies within the continuous temporal gaze signal. This perspective is motivated by the assumption that correct selections are characterized by relatively stable and consistent gaze patterns, whereas erroneous selections often involve increased uncertainty, corrective movements, or unstable fixations. As a result, errors can manifest as deviations from typical gaze dynamics. By treating errors as rare, distinct events, we align our methodology with established techniques in time series anomaly detection, as surveyed by Schmidl et al. (2022). In our previous work, we demonstrated the feasibility of applying anomaly detection to identify erroneous gaze-based selections (Severitt et al., 2025a). However, this prior work focused on offline analysis and did not address the integration of anomaly detection into real-time interaction. In this study, we extend that foundation by implementing and evaluating an anomaly-based error prevention system in a real-time VR environment and by investigating its impact on user performance and experience across different gaze-based interaction methods.
3 Methods
As shown in prior work, gaze behaviour surrounding selection events differs systematically between correct and incorrect selections (Severitt et al., 2024b). In particular, incorrect selections often deviate from the gaze patterns observed during correct selections. These deviations can be interpreted as anomalies in the gaze signal, which can be detected and used to prevent incorrect selections (Severitt et al., 2025a). Building on this idea, our EPS is based on anomaly detection to identify and suppress anomalous selections in real time.
3.1 Anomaly detection model
To distinguish between correct and incorrect selections, we model typical gaze behaviour associated with correct selections using an anomaly detection approach. For this purpose, we employ a TCNAE, which captures temporal dependencies in gaze dynamics and provides a reconstruction-based measure of selection validity, where deviations from typical patterns, resulting in a larger reconstruction error, indicate potential errors. In the following, we describe the model architecture, input representation, and training procedure in detail.
3.1.1 Method
The anomaly detection method we used to find anomalies in gaze behaviour in a short temporal window around the selection event is an AE based on a TCN implemented with PyTorch (Paszke et al., 2019). The architecture is shown in Figure 1. The idea is to train the network to reconstruct gaze patterns associated with correct selections. Non-normal selections—in this case, anomalies—are therefore never trained for reconstruction, which should result in a higher reconstruction error (Schmidl et al., 2022). To determine whether a selection was consistent with the gaze patterns observed during correct selections, we computed the reconstruction error e r r ∈ R between the input i n ∈ R 36 and the corresponding output of the network o u t ∈ R 36. Based on this error, the selection validity s ∈ { correct , incorrect } was determined as follows:
s = correct if e r r < t h incorrect if e r r ≥ t h

Here, t h ∈ R denotes a fixed threshold for the reconstruction error. This follows from the fact that gaze velocity tends to be low and stable during correct selections, whereas incorrect selections are characterized by more irregular gaze behaviour, resulting in higher reconstruction errors.
3.1.2 Input
In our previous work, we used angular distances between consecutive gaze vectors as input (Severitt et al., 2025a). This approach is sensitive to variations in the temporal spacing between consecutive samples, for instance due to dropped frames or varying sampling rates. Since the network operates on fixed-length input sequences, such variations can distort angular distances. To mitigate this issue, we use angular velocity, which normalizes the angular change by the time difference between samples and is therefore more robust to irregular sampling intervals. In practice, we used the last 37 gaze vectors, denoted as g v i ∈ R 3 with i = 0 , … , 36, captured in a temporal window centered around the selection event. At a sampling rate of 90 Hz, this window spans approximately 200 m before and after the selection, assuming no dropped frames. The temporal window was chosen to capture gaze behaviour immediately before and after the selection event. Following the selection, the window is limited to approximately 200 m to enable rapid detection, as human corrective responses typically occur with a delay of around 300 m. Using a shorter interval allows the system to respond earlier without relying on explicit user correction. Including gaze data before the selection provides additional context, ensuring that the model captures the transition into the selection event. Using these vectors, we computed a time series of angular velocities v j ∈ R for j = 0 , … , 35, defined as:
a j = 180 π arccos g v j ⋅ g v j + 1 with ‖ g v i ‖ = 1 ∀ i = 0 , … , 36 v j = a j t j + 1 − t j ⋅ 1 0 − 3
Here, a j represents the angular distance in degrees between consecutive normalized gaze vectors, and t j denotes the timestamp of g v j in milliseconds. The angular velocity v j is expressed in degrees per second, with the resulting profiles shown in Figure 2.

3.1.3 Training
To train the network, we use data from our previous experiment (Severitt et al., 2025b), using the same virtual environment described in subsection 3.2. We captured all correct selection events from the 52 participants and collected the necessary gaze vectors and timestamps surrounding the selection event (4,842 for Dwell Time, 4,725 for Head and Gaze, and 4,926 for Nod). With this data, we trained the network to reconstruct correct selections as described above. To find the hyperparameters, we performed five-fold cross-validation, following the procedure of our proof-of-concept study (Severitt et al., 2025a). The five-fold cross-validation was performed at the participant level. Thus, all samples from a given participant were assigned to the same fold, preventing overlap of participant-specific gaze behaviour between training and validation data. Cross-validation was performed separately for each interaction method, and the results were then aggregated across methods to determine the hyperparameter configuration that achieved the best overall performance. We used Mean Squared Error (MSE) as the loss function, which will also be used later as the reconstruction error. We trained the network for 2000 epochs with a batch size of 4,000, a learning rate of 1 0 − 2, and using the Adam optimizer. A separate network was trained for each interaction method to account for differences in gaze behaviour across techniques. All other hyperparameters were kept identical across networks. The threshold was computed by evaluating the MSE for all correct selections in the training dataset and selecting the 95th percentile of these values. The threshold was defined as the 95th percentile of the reconstruction errors obtained from the correct selections in the training set. This value was selected based on our previous work (Severitt et al., 2025a) and represents a compromise between detecting anomalous selections and limiting false rejection of correct selections.
3.2 Game
To evaluate the EPS for each interaction method, we integrated it into the VR game environment developed in our previous study (Severitt et al., 2025b). We only detail briefly aspects of the game that are relevant to understanding the rest of the paper, though, for more detail, see the previously mentioned study. Participants were instructed to achieve the highest possible score.
3.2.1 Game environment
The Virtual Environment (VE) consisted of a square-shaped room with interactive information displayed on the walls. One wall featured a high score leaderboard to motivate participants. Another wall displayed the current score in the center, along with a round counter positioned in the top right corner. A third wall showed a countdown timer for the active round. The overall layout was designed to provide continuous feedback and maintain user engagement, and is shown in Figure 3.

3.2.2 Gameplay
The core mechanics of the game involve identifying a target among flying robots positioned in the virtual environment. At the beginning of each round, robots move to one of approximately 18 predefined positions arranged within a 180-degree field in front of the user. Once a robot reaches its position, a sphere appears in front of it, displaying a character: the target is indicated by the letter “C”, while distractors are indicated by the letter “O”. Participants must identify and select the correct target using the gaze-based selection method assigned to the current experimental block. If a target is selected, the corresponding action is initiated that results in the destruction of the selected target. Destroying the correct target incurs no penalty, whereas destroying a distractor results in a deduction from the score.
When the EPS is active, selections are additionally validated using the TCNAE. The validation process is completed while the action is being executed and before task-relevant consequences, such as target destruction and score updates, occur. Model inference requires less than 1 m (Severitt et al., 2025a) and is therefore negligible compared to the approximately 200 m post-trigger acquisition period required to complete the TCNAE input window. If the selection is classified as anomalous, the action is interrupted before completion.
After a correct target has been destroyed, the robots reposition themselves by flying to a new position of the predefined locations, and the process repeats itself. Each round lasts 30 s, during which participants attempt to identify as many targets as possible. The experiment consists of three blocks, each corresponding to a different gaze selection method.
3.2.3 Scoring system
Participants received points based on their performance according to the following rules. Successfully destroying the correct target awarded between 5 and 20 points, depending on the response time. If the target was destroyed after more than 10 s, the minimum of 5 points was awarded. For faster responses, the score was computed using linear interpolation, with shorter reaction times yielding higher point values up to a maximum of 20. Incorrect selections—specifically destroying a distractor—resulted in a penalty of 21 points. This scoring scheme was designed to encourage both accuracy and speed, introducing time pressure to promote focused and efficient behaviour.
3.2.4 Gaze-based selection methods
Three gaze-based selection methods were compared in the study. The first method, Dwell time, allows participants to select a target by fixating on it for a specified duration, as originally introduced by Jacob (1990). The second method, Gaze and Head, requires both the gaze direction and head orientation to remain aligned with the target for the same duration, following the approach proposed by Sidenmark and Gellersen (2019). The third method, Nod, involves a gaze fixation followed by a nod gesture to confirm the selection, based on the work of Špakov and Majaranta (2012). The dwell time for the Dwell Time method was set to 0.4 s and for Gaze and Head to 0.3 s. Since the anomaly detection models were trained independently for each selection method, we refer to these three input techniques as distinct conditions in the context of the anomaly detection results.
3.3 Study
In this subsection, we provide details on the study design, including participant demographics, VR hardware and software environment, and overall experimental setup.
3.3.1 Participants
A total of 41 participants from the University of Tübingen and the surrounding area were recruited for this study; four participants dropped out due to technical issues. The remaining participants had a mean age of 26.14 ± 4.74 years. Of the remaining participants, 26 were women (mean age 26 ± 4.05 years) and 11 were men (mean age 26.45 ± 6.07 years). Most participants reported little or no prior experience with ET or VR. Further details on participant characteristics are provided in Table 1. This study was reviewed and approved by Faculty of Medicine at the University of Tübingen with a corresponding ethical approval identification code 247/202BO2. Participants provided their written informed consent to participate in this study.
TABLE 1. Participant demographics and self-reported experience. The table summarizes age, use of visual aids, and prior experience with Virtual Reality (VR) and Eye Tracking (ET), separated by gender.
| Topic | Category | General | Women | Men |
|---|---|---|---|---|
| Count | Count | 37 | 26 | 11 |
| Age | Mean age | 26.14 | 26 | 26.45 |
| Std age | 4.74 | 4.05 | 6.07 | |
| Minimal age | 19 | 20 | 19 | |
| Maximal age | 39 | 33 | 39 | |
| Visual aids | Contact lenses | 3 | 3 | 0 |
| Glasses | 13 | 7 | 6 | |
| No | 21 | 16 | 5 | |
| VR experience | Expert | 0 | 0 | 0 |
| A lot | 4 | 1 | 3 | |
| Moderate | 5 | 3 | 2 | |
| Little | 17 | 13 | 4 | |
| Not at all | 11 | 9 | 2 | |
| ET experience | Expert | 0 | 0 | 0 |
| A lot | 1 | 1 | 0 | |
| Moderate | 4 | 1 | 3 | |
| Little | 21 | 15 | 6 | |
| Not at all | 11 | 9 | 2 |
3.3.2 VR setup
The experiment was done with an HTC Vive Pro Eye head-mounted display (HMD; HTC Corporation, Taoyuan, Taiwan), which features a built-in Tobii eye tracker (Core SW 2.16.4.67) with an estimated accuracy of 0.5 ° to 1.1 ° and a sampling frequency of 120 Hz. ET data was calibrated and accessed using the Vive SRanipal SDK (HTC Corporation, Taoyuan, Taiwan). To extract gaze data from the VR headset, we employed the ZERO-Interface (Hosp and Wahl, 2023), which is integrated into the VisionaryVR (Hosp et al., 2024) framework. This interface provides separate three-dimensional gaze vectors for each eye, as well as a combined gaze vector. The data were accessible in real time and used for both gameplay and executing the EPS. Although the eye tracker samples gaze data at 120 Hz, the VR application accessed gaze data once per rendered frame (90 Hz). At each frame, the most recently available gaze sample was used. Consequently, the gaze data used by the EPS and for model training were effectively sampled at approximately 90 Hz. Additionally, all gaze data were recorded for post hoc analysis. An HTC Vive Pro Controller 2.0 was used for handheld interactions, such as adjusting sliders to complete questionnaires.
3.3.3 Design
At the beginning of each condition, participants were presented with a virtual text panel inside the VR environment, explaining how the respective gaze-based interaction method worked. This was followed by a training round using the method without the EPS, allowing participants to become familiar with the interaction technique. After the training round, another virtual instruction panel appeared, informing the participant that the next round would include the EPS. Upon completing this second round, a new panel prompted the participant with the question: “In which round was the error prevention system used?” Participants were required to select either First or Second. Following the training phase, the main experiment began. Participants completed five blocks, each consisting of two rounds—one with and one without the EPS. The order of the conditions within each block was randomized. After each block, participants again answered the same question to indicate which round they believed included the EPS. This served as a manipulation check to assess whether the system was noticeable. After completing the five blocks for a given interaction method, participants were transported to a new virtual scene, where they completed a questionnaire using VR-based User Interface (UI) elements. This entire procedure was repeated for all three gaze-based interaction methods, with the method order randomized across participants to counterbalance potential learning effects.
3.4 Measurements
In order to comprehensively evaluate the effects of the EPS, we collected both subjective and objective data. Subjective measurements focused on participants’ perceptions and experiences, while objective measurements captured task performance and system accuracy.
3.4.1 Subjective measurements
We collected data using a questionnaire that was completed after each method. The questionnaire evaluated six aspects of the EPS: usefulness (reduction of accidental selections), confidence (deletion of incorrect selections only), frustration (annoyance caused by the system), confidence boost (increased confidence when using gaze-based interaction), interference (impairment of intended actions), and preference (willingness to use the system in future gaze-based interfaces). All items were rated on a 10-point scale bounded by opposite statements. The complete questionnaire can be found in the appendix. At the end of the experiment, participants additionally indicated their preferred method (Gaze Dwell, Gaze and Head, or Nod) and whether they found the EPS helpful (yes, no, not sure).
3.4.2 Objective measurements
This was done in two ways. First, the performance of the participants was evaluated by comparing the conditions with and without the EPS for each method. Performance was evaluated based on the total number of points achieved and the number of incorrect selections. The points measure integrates both speed and accuracy, as faster correct selections result higher scores while incorrect selections result in a penalty. Next, the performance of the EPS itself was analyzed. This evaluation considered classification performance, reported separately for correct and incorrect selections. Specifically, we report the proportion of correct selections that were classified as correct, and the proportion of incorrect selections that were classified as incorrect.
3.5 Data analysis
Given the non-normal distribution of the data and the within-subjects design, non-parametric methods were applied. For subjective questionnaire data, Friedman tests were used non-parametric alternative to repeated-measures ANOVA (Friedman, 1937) to compare ratings across the three interaction methods (Dwell Time, Gaze and Head, Nod). Each questionnaire item was analyzed separately, treating interaction method as a single within-subject factor. For pairwise comparisons between conditions with and without the EPS, the Wilcoxon signed-rank tests were performed, which provide a non-parametric equivalent to the paired-samples t-test (Wilcoxon, 1945). To control for Type I error inflation in cases of multiple comparisons, the Bonferroni correction was applied (Dunn, 1961). Eta-squared ( η 2 ) was used as a measure of effect size in the Friedman test; for pairwise Wilcoxon comparisons, the effect size r was calculated based on a z-transformed approximation of the p-value (Cohen, 2013). All statistical analyses were conducted in Python (version 3.13.5) using NumPy (Harris et al., 2020), pandas (Wes, 2010), SciPy (Virtanen et al., 2020), and statsmodels (Seabold and Perktold, 2010). In addition, matplotlib (Hunter, 2007) and seaborn (Waskom, 2021) were used for data visualization.
4 Results
We present subjective and objective results of the study. Subjective results reflect participants’ perceptions of the EPS across the different interaction methods, while objective results report task performance and the effectiveness of the EPS.
4.1 Subjective measurements
The distribution of participants’ preferred interaction methods is shown in Table 2. In general, Gaze and Head was chosen most frequently, followed by Nod and Dwell Time. The evaluation of the EPS is summarized in Table 3, where the majority of participants reported that the system was helpful, while only a small number expressed uncertainty, and only one participant rated it as not helpful.
TABLE 2. Results of the post-experiment survey on preferred interaction methods, separated by gender and total numbers.
| Gender | Dwell time | Gaze and head | Nod |
|---|---|---|---|
| Women | 7 | 11 | 8 |
| Men | 4 | 5 | 2 |
| General | 11 | 16 | 10 |
TABLE 3. Results of the post-experiment survey on the perceived helpfulness of the EPS, separated by gender and total numbers.
| Gender | Yes | No | Not sure |
|---|---|---|---|
| Women | 20 | 1 | 5 |
| Men | 11 | 0 | 0 |
| General | 31 | 1 | 5 |
The participants’ ability to recognize whether EPS was active varied depending on the method: the proportion of correct identifications was 79.5 % for Dwell Time, 70.8 % for Gaze and Head, and 55 % for Nod, which is only slightly above chance level (50%). In Figure 4, the responses to the questionnaire for the different methods are shown. The analysis revealed significant differences between interaction methods for helpfulness ( χ 2 = 20.21 , p < 0.001 , η 2 = 0.289 ), trust ( χ 2 = 8.92 , p = 0.012 , η 2 = 0.127 ), confidence boost ( χ 2 = 9.88 , p = 0.007 , η 2 = 0.141 ), and preference ( χ 2 = 14.75 , p = 0.001 , η 2 = 0.211 ). Post-hoc pairwise comparisons showed that the Nod method received significantly lower ratings than Dwell Time and Head and Gaze across these dimensions. Descriptively, Dwell Time showed higher median ratings for helpfulness, confidence boost, and preference compared to the other methods, while Nod received the lowest ratings. No significant differences were found for frustration ( χ 2 = 0.16 , p = 0.925 , η 2 = 0.008 ), or disruption ( χ 2 = 0.23 , p = 0.893 , η 2 = 0.007 ), where median ratings were similar and generally low for frustration and disruption.

4.2 Objective measurements
The measurements of participant performance are shown in Figure 5. For the number of incorrect selections (summed over all rounds), significant differences were found across all three methods (see Figure 5a). With the EPS active, participants made markedly fewer incorrect selections. In the Dwell Time method, the median dropped from 22 without EPS to 1 with EPS ( W = 0.0 , p < 0.001 , r = 0.872 ). A similar effect was observed for Head and Gaze, with medians of 8 and 1, respectively ( W = 0.0 , p < 0.001 , r = 0.861 ). For Nod, the Wilcoxon signed-rank test also indicated a significant reduction in errors ( W = 22.5 , p = 0.009 , r = 0.387 ), although the median remained 0 in both conditions. This indicates that the reduction in errors is driven by participants who made errors, which is not reflected in the median. A comparable pattern was found for the points achieved (see Figure 5b). Participants gained substantially more points when the EPS was active. For Dwell Time, the median increased from 435 to 925 ( W = 703.0 , p < 0.001 , r = 0.872 ), and for Head and Gaze, from 745 to 819 ( W = 667.0 , p < 0.001 , r = 0.783 ). In contrast, no significant difference was observed for Nod, where the medians were nearly identical at 787 without EPS and 789 with EPS ( W = 409.0 , p = 0.196 , r = 0.140 ).

Table 4 summarizes the performance of the EPS across the three interaction methods. Overall, the EPS achieved high performance in classifying both correct and incorrect selections. Dwell Time and Head and Gaze showed a more balanced performance between correct and incorrect selections, whereas the Nod method had very high correct accuracy but lower accuracy for incorrect selections. Macro and micro metrics indicate consistent performance across methods, with differences in the trade-off between detecting incorrect selections and the acceptance of correct selections.
TABLE 4. Performance metrics of the EPS. Correct refers to the proportion of correct selections that were classified correctly, while Incorrect refers to the proportion of incorrect selections that were classified as incorrect. Macro is the mean of Correct and Incorrect, and micro represents the weighted mean based on the number of samples.
| Selection method | Correct | Incorrect | Macro | Micro |
|---|---|---|---|---|
| Dwell time | 0.794 | 0.928 | 0.861 | 0.829 |
| Gaze and head | 0.730 | 0.821 | 0.775 | 0.746 |
| Nod | 0.908 | 0.607 | 0.886 | 0.905 |
Figure 6 shows the performance of the EPS for each participant. Overall, the system performed well for most participants, with several achieving perfect accuracy. However, some participants exhibited lower performance, particularly in correct accuracy. The Gaze and Head method shows a higher variability, with more pronounced outliers compared to Dwell Time. It is also notable that for the Nod method, only a few participants made any incorrect selections while the EPS was active.

5 Discussion
In this study, we investigated the use of anomaly-based error prevention for gaze-based interaction in a VR visual search task. The results show that the EPS can effectively reduce incorrect selections and improve task performance while maintaining positive user perception. Importantly, the effectiveness of the system varies across interaction methods, highlighting that error prevention is strongly influenced by the underlying interaction technique. In the following, we discuss the key findings regarding participant performance, system effectiveness, and user perception.
The subjective results of the questionnaires show that the preferences of the participants largely corresponded to the results of our previous study without an EPS (Severitt et al., 2025b). Overall, participants rated the system positively, especially in the Dwell Time and Gaze and Head conditions. In contrast, perceived usefulness was lower for the Nod method, likely reflecting the already low baseline error rate in this condition. Consistent with this, participants found it more difficult to recognize whether the EPS was active when using Nod, whereas its presence was more apparent for the other methods. Interestingly, even participants with comparatively low accuracy scores stated that they appreciated the system. This suggests that occasional false rejections were not perceived as disruptive, as targets could be immediately reselected while visual attention remained focused on them. Nevertheless, maintaining user trust remains critical, as prior work has shown that trust can be difficult to restore once lost (Rittenberg et al., 2024).
The subjective results indicate that the EPS was generally well received and improved the experience of gaze-based interaction. Participants found it particularly helpful in situations with higher error rates, while its perceived usefulness was lower for the Nod method, where errors were already rare. These findings directly address our first research question: users appreciate and trust the EPS when it provides noticeable improvements without interfering with interaction. However, its perceived usefulness depends on the interaction method used.
Although the EPS substantially reduced incorrect selections, this benefit comes at the cost of rejecting a fraction of correct selections. This trade-off should be considered when evaluating the practical usefulness of anomaly-based error prevention systems. However, subjective assessments indicated a low level of impairment and frustration, suggesting that the participants did not find these rejections particularly stressful. Since rejected targets could be immediately reselected, the practical cost of a false rejection may be lower than indicated by the classification statistics alone. Future work should quantify reselection frequency and time costs more explicitly.
Previous work has shown that input errors can be detected within a short time frame after a selection event (Peacock et al., 2022), and that machine learning methods can effectively infer interaction intentions from gaze behaviour (Shevtsova et al., 2025). Our EPS combines these ideas by applying anomaly based classification in a real time setting shortly after selection and before task-relevant consequences occur. The results show that this approach is effective in practice: with the EPS enabled, participants made significantly fewer incorrect selections across all interaction methods. For Dwell Time and Gaze and Head, this reduction also translated into improved task performance, indicating that error prevention can provide benefits during interaction. Since incorrect selections directly reduced the score, improvements in score were partially expected whenever the EPS successfully prevented erroneous interaction outcomes. Therefore, score should be interpreted as an overall task-performance measure rather than as a direct measure of interaction efficiency. At the same time, the results highlight that the usefulness of such a system depends on the characteristics of the interaction method. The effect was limited for the Nod method, where incorrect selections were already rare. Consequently, there were fewer erroneous interaction outcomes that could be intercepted by the EPS, reducing the potential benefit of the system compared to Dwell Time and Gaze and Head.
Our EPS is based on a TCN-based architecture that has already proven itself in anomaly detection in time series data (He and Zhao, 2019). In our evaluation, the system achieved strong performance overall, but slightly below the levels reported in our previous proof-of-concept work (Severitt et al., 2025a), which can be attributed to the transition from offline analysis to a real-time setting. Acceptance rates for correct selections remained high across all interaction methods. The classification of incorrect selections was also strong, although less reliable for the Nod method, where the small number of errors limited meaningful evaluation. Notably, system performance varied across participants: while it was nearly perfect for some, it was less reliable for others. This variability suggests that individual gaze behaviour plays an important role in error detection, indicating that personal factors may influence how effectively the system can prevent incorrect selections.
These findings directly address our second research question: the EPS can be effectively applied in a real-time environment, reducing errors and improving task performance in interaction methods where errors are likely to occur. While its impact is smaller for methods with inherently low error rates, the system still works reliably within the time constraints of online interaction. These results support the use of anomaly detection based on gaze behaviour as a practical approach for real-time error prevention in gaze-based interfaces.
Overall, the results demonstrate that EPS can effectively reduce errors and improve performance in interaction methods that are more prone to incorrect selections while also being positively received by participants. However, its effectiveness varies across interaction methods and individuals. These differences suggest that a uniform approach may not be sufficient. Previous research has shown that gaze behaviour is highly individual and context-dependent (Hessels, 2020; gillet2021gazebehaviorrobot), highlighting the need for more adaptive approaches. Future iterations of the system could therefore benefit from dynamic adjustment of thresholds or detection strategies to better accommodate individual characteristics and interaction styles. Such adaptability has the potential to improve robustness across users while maintaining the positive user experience observed in this study.
6 Limitations
While the results demonstrate the effectiveness of the EPS in a controlled VR setting, there are several limitations to consider. First, the findings relate specifically to the visual search task performed and may not be directly transferable to other applications or interaction contexts. In real-world settings, gaze behaviour is often more variable, with frequent distractions and unintended fixations. This may particularly affect interaction techniques such as Dwell time, which rely on stable fixations and may therefore be more susceptible to false activations outside controlled environments. Second, the evaluation was conducted exclusively in a VR using a single application. Transferring the approach to other contexts, such as alternative task types or augmented reality, may require further adjustments to both the interaction design and the underlying detection model. In particular, adapting the system to new environments will likely require additional training data to ensure robust performance across different conditions. Future work should therefore validate the system in a broader range of environments and interaction scenarios to assess its generalizability.
Another limitation of the present study concerns the distinction between selection correctness and user intention. Selections of the target were labeled as correct, whereas selections of distractors were labeled as incorrect. Consequently, the anomaly detection model was trained to distinguish gaze patterns associated with correct and incorrect selections rather than directly infer users’ intentions. Although participants were instructed and incentivized to select the target, the present data do not allow conclusions about the subjective intention behind individual selections. It therefore remains possible that some correct selections resulted from unintended activations, while some incorrect selections reflected intentional but mistaken choices. Future work should incorporate independent measures of intention to further investigate the relationship between user intention, selection correctness, and anomalous gaze behaviour.
7 Conclusion
This study demonstrates that real-time EPS based on anomaly detection can effectively support gaze-based interaction in VR environments. Regarding the first research question, participants generally perceived the system positively and reported that it was helpful and boosted their confidence. Regarding the second research question, the system succeeded in reducing errors and improving task performance in methods prone to incorrect selections, such as Dwell Time and Gaze and Head, while the benefits were limited in methods such as nod with few errors. The variability among participants underscores the need for adaptive approaches, and future work should evaluate the system in other tasks, interaction modalities, and environments outside of VR.
Generative AI statement
The author(s) declared that generative AI was used in the creation of this manuscript. The author(s) declared that generative AI was used in the creation of this manuscript. Copilot, DeepL and Quillbot were utilized to assist with grammatical and editorial refinement. All manuscript content was manually generated and strictly verified by the authors.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2026.1918264/full#supplementary-material
Supplementary material Supplementary material at the source
Footnotes
Edited by: Elda Cina, American University of the Middle East, Kuwait
Reviewed by: Xiaoxi Du, Southeast University, China
Rajashekhar V. S., Indian Institute of Technology Kanpur, India
Statements
- Conflict of interest
Authors YS, NC, and SW were employed by Carl Zeiss Vision International GmbH.
The remaining author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
- Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 101086206 PLACES, the institutional strategy of the University Tübingen (Deutsche Forschungsgemeinschaft, ZUK 63) and was done in an industry-on-campus-cooperation between the University Tübingen and Carl Zeiss Vision International GmbH. The author(s) declare that Carl Zeiss Vision International GmbH was not involved in the study design, collection, analysis, interpretation of data, the writing of this article, or the decision to submit it for publication.
- Data availability
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
- Ethics approval
The studies involving humans were approved by Faculty of Medicine at the University of Tübingen with a corresponding ethical approval identification code 247/202BO2. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
- Author contributions
BS: Conceptualization, Data curation, Formal Analysis, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review and editing. YS: Methodology, Validation, Writing – review and editing. NC: Methodology, Supervision, Writing – review and editing. SW: Funding acquisition, Project administration, Resources, Supervision, Writing – review and editing.
- Acknowledgements
We acknowledge support from the Open Access Publication Fund of the University of Tübingen.
References
- Bovo R., Binetti N., Brumby D. P., Julier S. (2020). “ Detecting errors in pick and place procedures: detecting errors in multi-stage and sequence-constrained manual retrieve-assembly procedures,” in Proceedings of the 25th International Conference on Intelligent User Interfaces, 536 – 545.
- Bruder C., Weber P., Hasse C. (2016). “ To look and (not) see: predicting the detection of automation failures based on the eye movements of human operators,” in Proceedings of the International Conference on Human-Computer Interaction in Aerospace (New York, NY, USA: Association for Computing Machinery). doi:10.1145/2950112.2964579
- Cohen J. (2013). Statistical Power Analysis for the Behavioral Sciences. Routledge. doi:10.4324/9780203771587
- David-John B., Peacock C., Zhang T., Murdison T. S., Benko H., Jonker T. R. (2021). “ Towards gaze-based prediction of the intent to interact in virtual reality,” in ACM Symposium on Eye Tracking Research and Applications, 1 – 7.
- Dunn O. J. (1961). Multiple comparisons among means. J. American Stat. Assoc. 56, 52 – 64. doi:10.1080/01621459.1961.10482090
- Esteves A., Velloso E., Bulling A., Gellersen H. (2015). “ Orbits: gaze interaction for smart watches using smooth pursuit eye movements,” in UIST '15: Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (New York, NY, USA: Association for Computing Machinery) 15, 457 – 466. doi:10.1145/2807442.2807499
- Flook R., Shrinah A., Wijnen L., Eder K., Melhuish C., Lemaignan S. (2019). On the impact of different types of errors on trust in human-robot interaction. Interact. Stud. 20, 455 – 486. doi:10.1075/is.18067.flo
- Friedman M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. American Stat. Assoc. 32, 675 – 701. doi:10.1080/01621459.1937.10503522
- Gardony A. L., Okano K., Hughes G. I., Kim A. J., Renshaw K. T., Sipolins A. (2024). Characterizing information access needs in gaze-adaptive augmented reality interfaces: implications for fast-paced and dynamic usage contexts. Human–Computer Interact. 39, 553 – 583. doi:10.1080/07370024.2023.2260788
- Gemicioglu T., Winters R. M., Wang Y.-T., Gable T. M., Paradiso A., Tashev I. J. (2023). “ Gaze & tongue: a subtle, hands-free interaction for head-worn devices,” in Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3544549.3583930
- Harris C. R., Millman K. J., van der Walt S. J., Gommers R., Virtanen P., Cournapeau D. (2020). Array programming with NumPy. Nature 585, 357 – 362. doi:10.1038/s41586-020-2649-2
- He Y., Zhao J. (2019). Temporal convolutional networks for anomaly detection in time series. J. Phys. Conf. Ser. 1213, 042050. doi:10.1088/1742-6596/1213/4/042050
- Hessels R. S. (2020). How does gaze to faces support face-to-face interaction? A review and perspective. Psychonomic Bull. Rev. 27, 856 – 881. doi:10.3758/s13423-020-01715-w
- Hosp B. W., Wahl S. (2023). “ Zero: a generic open-source extended reality eye-tracking controller interface for scientists,” in Proceedings of the 2023 Symposium on Eye Tracking Research and Applications (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3588015.3589203
- Hosp B. W., Dechant M., Sauer Y., Severitt B., Agarwala R., Wahl S. (2024). Visionaryvr: an optical simulation tool for evaluating and optimizing vision correction solutions in virtual reality. Sensors 24, 2458. doi:10.3390/s24082458
- Hunter J. D. (2007). Matplotlib: a 2d graphics environment. Comput. Sci. Eng. 9, 90 – 95. doi:10.1109/MCSE.2007.55
- Istance H., Bates R., Hyrskykari A., Vickers S. (2008). “ Snap clutch, a moded approach to solving the midas touch problem,” in Proceedings of the 2008 Symposium on Eye Tracking Research & Applications (New York, NY, USA: Association for Computing Machinery), 221 – 228. doi:10.1145/1344471.1344523
- Jacob R. J. K. (1990). “ What you look at is what you get: eye movement-based interaction techniques,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery) 90, 11 – 18. doi:10.1145/97243.97246
- Mohan P., Goh W. B., Fu C.-W., Yeung S.-K. (2018). “ Dualgaze: addressing the midas touch problem in gaze mediated vr interaction,” in 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 79 – 84. doi:10.1109/ISMAR-Adjunct.2018.00039
- Padovani R. R., Ferreira L. N., Lelis L. H. S. (2019). “ Be inaccurate but don’t be indecisive: how error distribution can affect user experience,” in Proceedings of the AAAI Conference on Artificial Intelligence 33, 2604 – 2611. doi:10.1609/aaai.v33i01.33012604
- Parisay M., Poullis C., Kersten-Oertel M. (2021). Eyetap: introducing a multimodal gaze-based technique using voice inputs with a comparative analysis of selection techniques. Int. J. Human-Computer Stud. 154, 102676. doi:10.1016/j.ijhcs.2021.102676
- Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G. (2019). “ Pytorch: an imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems (Curran Associates, Inc.) 32, 8024 – 8035.
- Peacock C. E., Lafreniere B., Zhang T., Santosa S., Benko H., Jonker T. R. (2022). Gaze as an indicator of input recognition errors. Proc. ACM Hum.-Comput. Interact. 6, 1 – 18. doi:10.1145/3530883
- Rajanna V., Hammond T. (2016). “ Gawschi: gaze-augmented, wearable-supplemented computer-human interaction,” in Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications (New York, NY, USA: Association for Computing Machinery), 233 – 236. doi:10.1145/2857491.2857499
- Rittenberg B. S. P., Holland C. W., Barnhart G. E., Gaudreau S. M., Neyedli H. F. (2024). Trust with increasing and decreasing reliability. Hum. Factors 66, 2569 – 2589. doi:10.1177/00187208241228636
- Schmidl S., Wenig P., Papenbrock T. (2022). Anomaly detection in time series: a comprehensive evaluation. Proc. VLDB Endow. 15, 1779 – 1797. doi:10.14778/3538598.3538602
- Schweigert R., Schwind V., Mayer S. (2019). “ Eyepointing: a gaze-based selection technique,” in Proceedings of Mensch Und Computer 2019 (New York, NY, USA: Association for Computing Machinery) 19, 719 – 723. doi:10.1145/3340764.3344897
- Seabold S., Perktold J. (2010). “ Statsmodels: econometric and statistical modeling with python,” in 9th Python in Science Conference.
- Sendhilnathan N., Zhang T., Lafreniere B., Grossman T., Jonker T. R. (2022). “ Detecting input recognition errors and user errors using gaze dynamics in virtual reality,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3526113.3545628
- Sendhilnathan N., Zhang T., Bethge D., Nebeling M., Grossman T., Jonker T. R. (2025). “ A multimodal approach for targeting error detection in virtual reality using implicit user behavior,” in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3706598.3713777
- Severitt B., Castner N. J., Lukashova-Sanz O., Wahl S. (2023). “ Leveraging gaze for potential error prediction in ai-support systems: an exploratory analysis of interaction with a simulated robot,” in Companion Publication of the 25th International Conference on Multimodal Interaction (New York, NY, USA: Association for Computing Machinery), 56 – 60. doi:10.1145/3610661.3617163
- Severitt B. R., Castner N., Wahl S. (2024a). “ Bi-directional gaze-based communication: a review,” in Multimodal Technologies and Interaction 8. doi:10.3390/mti8120108
- Severitt B. R., Lenhart P., Hosp B. W., Castner N. J., Wahl S. (2024b). “ Communication breakdown: gaze-based prediction of system error for ai-assisted robotic arm simulated in vr,” in Proceedings of the 2024 Symposium on Eye Tracking Research and Applications (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3649902.3653339
- Severitt B. R., Sauer Y., Castner N. J., Fuhl W., Wahl S. (2025a). “ Recognition of errors in gaze-based interaction with anomaly detection,” in Proceedings of the 2025 Symposium on Eye Tracking Research and Applications (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3715669.3725895
- Severitt B. R., Sauer Y., Neugebauer A., Agarwala R., Castner N., Wahl S. (2025b). The interplay of user preference and precision in different gaze-based interaction methods in virtual environments. Front. Virtual ReaI. 6. doi:10.3389/frvir.2025.1576962
- Shevtsova Y. G., Yashin A. S., Shishkin S. L., Vasilyev A. N. (2025). Disentangling high-paced alternating i/o in gaze-based interaction. IEEE Access 13, 81948 – 81966. doi:10.1109/ACCESS.2025.3568650
- Sidenmark L., Gellersen H. (2019). “ Eye&head: synergetic eye and head movement for gaze pointing and selection,” in Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New York, NY, USA: Association for Computing Machinery), 1161 – 1174. doi:10.1145/3332165.3347921
- Sidenmark L., Potts D., Bapisch B., Gellersen H. (2021). “ Radi-eye: hands-free radial interfaces for 3d interaction using gaze-activated head-crossing,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1 – 11.
- Špakov O., Majaranta P. (2012). “ Enhanced gaze interaction using simple head gestures,” in Proceedings of the 2012 ACM Conference on Ubiquitous Computing (New York, NY, USA: Association for Computing Machinery) 12, 705 – 710. doi:10.1145/2370216.2370369
- Velloso E., Wirth M., Weichel C., Esteves A., Gellersen H. (2016). “ Ambigaze: direct control of ambient devices by gaze,” in Proceedings of the 2016 ACM Conference on Designing Interactive Systems (New York, NY, USA: Association for Computing Machinery) 16, 812 – 817. doi:10.1145/2901790.2901867
- Virtanen P., Gommers R., Oliphant T. E., Haberland M., Reddy T., Cournapeau D. (2020). SciPy 1.0: fundamental algorithms for scientific computing in python. Nat. Methods 17, 261 – 272. doi:10.1038/s41592-019-0686-2
- Waskom M. L. (2021). Seaborn: statistical data visualization. J. Open Source Softw. 6, 3021. doi:10.21105/joss.03021
- Wes M. K. (2010). “ Data structures for statistical computing in python,” in Proceedings of the 9th Python in Science Conference. Editors van der Walt S., Millman J., 56 – 61. doi:10.25080/Majora-92bf1922-00a
- Wilcoxon F. (1945). Individual comparisons by ranking methods. Biom. Bull. 1, 80 – 83. doi:10.2307/3001968
Citation
Severitt, B. R., et al. (2026). A real-time error prevention system for gaze-based interaction in virtual reality based on anomaly detection. Frontiers in Virtual Reality, Volume 7. https://doi.org/
- Received
- Accepted
- Published
- Volume
- 7 - 2026
Copyright
Copyright © 2026 Severitt, Sauer, Castner and Wahl.
License: https://creativecommons.org/licenses/by/4.0/
*Correspondence: Björn R. Severitt
Article metrics
- Views
- 0
- Downloads
- 0
- Citations
- 0
- Shares
- 0
