Integrating real-world situation awareness cues in virtual reality to enhance user wellbeing

1 Introduction

Virtual reality (VR) devices are currently typically designed to support users’ safety, productivity, and enjoyment (Bartlett et al., 2018; Damiani et al., 2018; Dani, 2019). However, as our relationship with VR technology continues to deepen, the broader issues of wellbeing need to be put front and center. The higher-level relation between wellbeing and common digital technologies has been studied extensively (Forsman et al., 2018; Lee et al., 2019; Yang and Li, 2021). However, there remains a lack of research on wellbeing specifically in the context of VR (Adhyaru and Kemp, 2022; Johnson et al., 2013), despite the known effects of VR use on wellbeing (Gotsis, 2009; Pallavicini and Pepe, 2020; Siricharoen, 2019).

This paper views wellbeing in VR from the lens of positive computing, which comprises multiple dimensions such as positive emotions, empathy, and motivation (Calvo and Peters, 2014). This work particularly focuses on the mindfulness dimension, which is defined as the state of being attentive and aware of what is happening at the present moment (Cardaciotto et al., 2008; Hill and Labbé, 2014).

VR systems are exceptional at immersing the user into the virtual environment, effectively blocking out the immediate external (i.e., “real”) world. This immersion, while beneficial for the VR experience, can make it difficult for users to maintain a mental representation of what is happening in the real world, placing the burden of acquiring this awareness on the user (Li et al., 2020; O’Hagan et al., 2022). Providing Situation Awareness (SA) cues about the real world from within the virtual environment has been proposed as a potential remedy for this lack of awareness (Fereydooni and Walker, 2020).

However, the effect of mechanisms designed to bridge the real world and the virtual environment on user perception and their VR experience has not been investigated (George et al., 2019; Medeiros et al., 2021). Keeping users in the loop about the dynamic real world through virtual cues can prevent them from searching for this information themselves, thus freeing up their processing resources and leading to a lower perceived workload (Tsang and Vidulich, 2006). However, potential drawbacks of providing cues include the perception that the cues are distracting. The effectiveness and optimal design of these cues likely vary depending on the specific real-world environment and the demands of the VR task. Understanding how users perceive and process the information conveyed by these cues, and how the structure or information architecture of the presentation impacts their experience, remains a significant area for investigation.

This paper reports on two studies investigating the impact of providing real-world awareness cues within VR, specifically in a home environment. Study 1 explores users’ perception of different types of SA cues and how their need for information about the real world influences their experience, including perceived workload, situation awareness, VR presence, and positive emotions. It also examines the effects of cue persistence and whether cues delivered via a “Push” (appearing automatically) or “Pull” (accessed on demand) mechanism are preferred. The research questions addressed by Study 1 were:

RQ1.1. How does users’ need to attend to real-world tasks while immersed in VR influence their experience, in terms of their perceived workload, and perceived situation awareness, VR presence, and physiological responses, specifically gaze behavior?

RQ1.2. How does providing cues about the home environment in the virtual environment affect users’ experience?

RQ1.3. How does the persistence of cues affect users’ experience?

RQ1.4. How does users’ autonomy in receiving cues affect their experience?

Study 2 delves into the information architecture of SA cues, comparing the effects of different display types, specifically Textual vs. Embedded pictorial representations, on user experience, perceived workload, situation awareness, and gaze behavior. This study also investigates the interplay between cue design and the activity level of the VR task (Active vs. Passive). The research questions addressed in Study 2 were:

RQ2.1. How do different information displays in VR influence users’ experience, in terms of their perceived workload, and perceived situation awareness about the real world, VR presence, and physiological responses, specifically gaze behavior?

RQ2.2. How does the activity level of the tasks that users engage with in VR influence the effect of the information displays?

Collectively, these two studies aim to enhance our understanding of how best to integrate real-world awareness into immersive VR experiences in a home setting. By examining user perceptions, the influence of information need, and the impact of different information architectures, this research provides insights into designing less distracting and more effective SA cues that can support users’ mindful attention to both the virtual and physical worlds, ultimately contributing to their overall wellbeing in VR.

2 Background

2.1 Wellbeing in VR

Wellbeing encompasses a dynamic interplay of personal, social, and transcendent factors (Calvo and Peters, 2014). This multifaceted nature emphasizes the necessity of examining VR experiences through a holistic lens, taking into account not only the immersive qualities of VR environments but also their potential impact on users’ mental and emotional states. The positive computing framework (Calvo and Peters, 2014) identifies several key dimensions of wellbeing relevant to VR, including positive emotions, motivation and engagement, and mindfulness. Each dimension offers unique insights into how VR technology can shape users’ experiences and influence their overall sense of wellbeing. By understanding and addressing these dimensions, VR developers and researchers can design experiences that promote positive outcomes and mitigate potential risks, ultimately enhancing users’ experiences in virtual environments.

Mindfulness—the main focus of this paper and one of the wellbeing dimensions in the positive computing framework—consists of a non-reflective and non-judgmental observation of one’s thoughts, feelings, bodily sensations, and the surrounding environment (Kabat-Zinn, 2015). It would be difficult for people to be mindful and pay attention to a VR task when they are using the VR headset in a dynamic real-world environment. This situation is not really a question of task switching, since the user knows that the real-world task continues (i.e., the co-workers or family are still asking for things). The user needs to devote some attention and memory to maintaining a representation of the external task while in the virtual environment, and vice versa. This divided attention is one of the challenges of leaving the real world and getting immersed in a virtual environment (Farve et al., 2016; Jones et al., 2014). The requirement to engage with multiple sources of information that are separated spatially or temporally can result in cognitive overload (Ayres and Cierniak, 2012; Kirsh, 2000). Whereas divided attention and multitasking are culturally viewed as a positive expression of productivity, they lead to reduced performance (Adler and Benbunan-Fich, 2012; Kc, 2014), and are shown to have a negative correlation with wellbeing (Baethge and Rigotti, 2013; Claessens et al., 2004; Robinson and Smallman, 2006; Zijlstra et al., 1999). The solution to this issue may be achieved by reducing users’ urge to take off the headset to peek outside the virtual environment, or to allocate some of their attention resources to the real-world environment (e.g., listening for the doorbell, even if for a short while).

2.2 VR at home

VR devices offer a multitude of advantages for various tasks, including work, entertainment, and socializing. In terms of work, the immersion provided by these devices can enhance job performance, motivation, and engagement by creating an optimal work environment, free from distractions present in the real world (Barathi et al., 2018; Mestre et al., 2011). VR provides a unique opportunity to reduce interruptions while working, and then facilitate unplugging from work at the end of the day. VR can also facilitate collaboration among remote teams, as individuals can interact as if they were in the same room, fostering teamwork and idea sharing (Çoban and Goksu, 2022). VR also facilitates training and skill development by simulating real-world scenarios, enabling users to practice and refine their skills in a safe and controlled environment (Joshi et al., 2021; Norris et al., 2019). In terms of entertainment, VR devices offer a revolutionary way to experience media, allowing users to feel like they’re part of the action in games, movies, and virtual experiences (Dani, 2019). Furthermore, VR can provide unique opportunities for socializing, allowing people to connect and interact in virtual spaces regardless of physical distance, which is especially valuable for individuals who may have difficulty socializing in traditional settings (Maloney and Freeman, 2020; Maloney et al., 2021).

At the same time, the occluded view introduced by the device and the lack of situation awareness about the dynamic real-world environment may influence users’ virtual experience and mindful attention to the VR task. Although there have been many studies developing various virtual experiences for home use, the study of the implications of this adoption on users’ experience is lacking. This paper is a step towards addressing this gap.

2.3 Situation awareness in VR

As mentioned before, VR users get fully immersed in VR such that they lose SA. In many cases, this reduction of external SA is considered beneficial, in that it can support further immersion in the virtual environment. However, depending on the VR task, the lower SA may also induce stress, increase cognitive load, and negatively affect users’ decision making (Kirsh, 2000) which in turn influence users’ attention, immersion, and wellbeing. Understanding users’ perceptions and cognitive assessments in using VR devices is crucial for successful integration of VR in dynamic real-world environments which are becoming increasingly prevalent. In an optimal immersion scenario, the user is provided with just enough information about the real world, in an appropriate method, such that they don’t have to set aside attentional resources to gain SA, and that information does not interfere with their VR experience.

VR users often desire to receive information about the real world while being immersed in the virtual environment (Ghosh et al., 2018; O’Hagan et al., 2022), and some studies have attempted to address this issue; for instance, users may have a fear of colliding into real-world objects while moving in the virtual environment, whether the actual collision risk exists or not. Some proposed mechanisms such as Oculus Guardian (Oculus, 2020) and HTC VIVE’s play area (Casterson, 2016) notify users about surrounding physical objects so they do not collide with them. Another mechanism is Reality Check (Hartmann et al., 2019), which merges virtual and physical environments, enabling users to be aware of their surroundings while immersed in the virtual environment. In another proposed solution (Kang and Han, 2020), the device notifies the user of upcoming obstacles while they are walking. Other approaches include redirecting users’ walking paths to prevent them from colliding with static objects (Kanamori et al., 2019; Sousa et al., 2019) or with moving people (Sun et al., 2018) in the real world. Other solutions for providing users with information about the real world include using cues to notify VR users about non-immersed bystanders in socially related, non-critical interaction contexts (Medeiros et al., 2021; O’Hagan and Williamson, 2020; O’Hagan et al., 2020; Simeone, 2016). Various modalities such as audio, haptics, and visual displays, have been investigated for communicating real-world information in the virtual environment (George et al., 2020; Ghosh et al., 2018; May et al., 2020; O’Hagan and Williamson, 2020).

Multiple studies have shown that some users are worried about the distraction and disruption that the cues may cause by inserting too much “reality” in VR (Fereydooni et al., 2022; O’Hagan et al., 2022). Although some participants find the cues to be an essential part of their experience, some express concern about being distracted by them (Fereydooni et al., 2022). Various factors may cause the SA cues to disrupt users’ presence in the virtual environment. For instance, Medeiros et al. (2021) briefly reported mixed user feedback regarding push and pull cues: some participants preferred that the VR environment pushed the real-world information to them so they did not have to actively check for information themselves, but others found this to be distracting and preferred to have more autonomy in pulling (i.e., requesting) information as needed.

Given the growing number of solutions bridging the real world and the virtual environment, it is important to consider the effect of these cues on various aspects of users’ virtual and real-world experiences such as workload and perceived risk.

2.4 Information displays

A variety of information displays have been developed to allow people to keep awareness of information unrelated to their main task; for example, workers may want to keep track of their favorite team’s score or the weather outside. Various features have been implemented to deliver this information to users; for instance, this may be through push or pull mechanisms. Push notifications involve the delivery of information from a server or central source to a user’s device without any explicit request from the user (Warren et al., 2014). This method is often used to provide timely updates or alerts about events, messages, or changes relevant to the user. For example, a push notification might inform a user about a new email, a social media update, or breaking news. Push notifications are valuable for keeping users informed and engaged, as they deliver information directly, even if the user is not actively using the app or device at the time (Wohllebe et al., 2021). On the other hand, pull notifications require the user to actively request or retrieve information from a server or source (Graham et al., 2006). Users initiate the process by actively seeking out updates or information by refreshing a webpage, opening an app, or executing a specific command. Pull notifications are commonly used in scenarios where users have control over when and how they access information, such as checking email, browsing social media feeds, or searching for specific content online. Unlike push notifications, pull notifications rely on user engagement and initiative to access information, making them more user-driven and less intrusive. Together, push and pull notifications contribute to a seamless user experience on smartphones by keeping users informed and allowing them to access relevant content at their convenience (Burgstahler et al., 2014; Prajitno et al., 2019). Study 1, here, explores the effect of these notifications in VR experiences.

Peripheral displays are frequently utilized to minimize the distraction of information, ensuring that it remains accessible without overwhelming the user’s primary focus. These information displays are predominantly designed for 2D environments like desktops, phones, etc. However, the implications of transferring these displays to VR environments have not been studied. The immersive nature of VR raises questions about how information can be effectively presented without detracting from the overall experience or causing disorientation. Study 2 investigates these displays to understand how these displays may be adapted and optimized for VR contexts, ensuring they enhance rather than disrupt user interaction and immersion.

3 Study 1: overview

To explore these aspects of cues, Study 1 examined users’ perception of the cues and how users’ need for information about the real world affected their workload, perceived situation awareness, and VR presence. As a reminder, the research questions were:

• RQ1.1 How does users’ need to attend to real-world tasks while immersed in VR influence their experience, in terms of their perceived workload, and perceived situation awareness, VR presence, and physiological responses, specifically gaze behavior?
• RQ1.2. How does providing cues about the home environment in the virtual environment affect users’ experience, in terms of their perceived workload, and perceived situation awareness, VR presence, and physiological responses, specifically gaze behavior?
• RQ1.3. How does the persistence of cues affect users’ experience, in terms of their perceived workload, and perceived situation awareness, VR presence, and physiological responses, specifically gaze behavior?
• RQ1.4. How does users’ autonomy in receiving cues affect their experience, in terms of their perceived workload, and perceived situation awareness, VR presence, and physiological responses, specifically gaze behavior?

4 Study 1: methodology

4.1 Participants

Study 1 included 72 participants recruited via the university’s undergraduate research participant pool. The age of participants varied between 18 and 32 years (M = 19.8, SD = 2.4), encompassing 32 females, 39 males, and 1 individual who preferred not to disclose their gender. Inclusion criteria comprised sufficient mobility to use a VR device, and normal or corrected-to-normal vision and hearing. Among all the participants, 16.7% owned a VR headset, and 23.6% stated that they use a VR device at least a few times per year.

4.2 Ethics statement

This study was reviewed and approved by the University Institutional Review Board (IRB). Participants were compensated with one hour of coursework extra credit for their participation in the hour-long study session. To guarantee completely voluntary participation and eliminate potential coercion, an equitable alternative non-research assignment was made available for students to earn the exact same credit amount within their respective coursework.

4.3 Experiment design

Study 1 was a 3 × 3 mixed design, studying the effect of two independent variables (Information Need, and Cue Type) on users’ perceived situation awareness, perceived workload, VR presence, and wellbeing. Information Need was a within-subject variable with three levels [VR only, Need SA-no cues (dubbed No-Cue), Need SA-cues (dubbed Cue)]. Cue Type was a between-subjects variable with three levels (push temporary, push persistent, pull temporary). Each participant was randomly assigned to a Cue type condition and performed the experimental tasks in three rounds to experience the three levels of the Information Need. The order of the conditions was counterbalanced using a Latin Square Design.

4.4 VR experience

The VR experience was a passive entertainment task, without the need for the participants to actively interact with the virtual content. It consisted of watching 360 VR videos of aerial tours of three different cities, each presented during a participant’s three trials. The aerial tours were taken from AirPano VR and VR Gorilla YouTube channels (AirPano, 2021; Gorilla, 2021) and were adapted for VR. To establish clear asset provenance and comply with non-commercial academic evaluation provisions, these public-facing media streams were individually extracted, optimized for resolution, and reformatted. The equirectangular projection streams were mapped as individual, isolated texture layers projected onto an inverted skybox architecture within the virtual ecosystem, ensuring no unauthorized modification or commercial redistribution of the original cinematography occurred.

The order of the VR tours was fixed; Angkor Wat was shown during the first round, Las Vegas during the second round, and Bangkok during the third round. During the tours, participants were walked or flown through each city and listened to a narrative highlighting different landmarks. Each tour was about 7 min long to minimize the buildup of cyber-sickness symptoms and fatigue (Davis et al., 2015; Kennedy et al., 2000).

The VR task was developed in Unity 2019.1.2 and was presented on an HTC Vive Pro Eye, a tethered VR headset connected to a PC to run the application (HTC Vive, 2021). The individual pipeline for asset integration was specifically structured to maintain compatibility with this legacy Unity 2019 camera rig configuration. This headset is equipped with built-in Tobii eye trackers which feature a sampling rate of 120 Hz and a manufacturer-reported spatial accuracy of 0.5∘–1.1∘ Tobii (2023).

4.5 Research context

The study took place at a 2-story research facility that is designed to mimic a suburban house to provide an authentic home environment. Each participant sat at the dining room table in the second floor of the research facility (see Figure 1).

In all three rounds of the study, participants could hear sounds coming from the home environment, indicating real-world events including phone notifications, doors opening/closing, and door bell ringing. These sounds were recorded, and played at specific moments in the study via surround speakers. The first round consisted of 3 real-world events, and the second and third rounds consisted of 4 real-world events. In the VR-Only condition, participants were asked to disregard these events and only focus on VR. In the Cue and the No Cue conditions, they were asked to keep track of the real-world events. In the Cue condition, participants received visual cues about the real world events while they were immersed in the virtual world. However, participants were not given any further information about the real world events in the No-Cue condition.

4.6 SA cues

Each cue followed the user’s line of gaze to ensure they would see the cues. The cues were positioned at the lower middle of user’s field of view to provide information without blocking their view of the video.

These SA cues consisted of information about who entered the house, what group chat received a message, and why the doorbell rang. Push Temporary cues consisted of a short text explaining the sound coming from the real-world home environment, and appeared every time something happened in the house environment. These cues stayed on screen for 5 s to allow enough time for the user to read the text (Card et al., 2018). Pull Temporary cues looked the same as Push Temporary cues, but the participant could make them appear and disappear using the controller. Push Persistent cues stayed on the screen all throughout the VR video. An additional line appeared on top of the previous lines upon a new real-world event. Figures 2 and 3 display an example of the types of SA cues.

4.7 Procedure

After the participant signed the informed consent form, she was screened for simulator sickness (Gable and Walker, 2013). The VR experience during the screening protocol consisted of a flight over the Siberian desert (AirPano, 2021). The survey software automatically scored the participants’ pre- and post-drive responses, leading to a recommendation for continuing or exiting the study. After passing the screening, the participant was calibrated for the eye tracking system. She was then provided with the real-world scenario that she had to imagine for the study; she was asked to imagine that she lives in the apartment with two roommates, and that she is waiting for her teammates to arrive at her apartment to work on a class project. She was told that food is going to be delivered to the house for the meeting, and that she is expecting a package in the mail. The participant was also told that she may receive notifications on her phone, including food delivery status and text from teammates and friends, and their phone is not on silent mode. Finally, she was asked to imagine that she decided to use a VR headset before her teammates arrive. The participant was informed that none of the the real-world home events are urgent and no action is required in response to them. This scenario was designed so that all participants were in a consistent at-home frame of mind. In the No Cue and Cue conditions the participant was also asked to keep track of these tasks while using VR. This requirement was in place to simulate users’ need for keeping track of real-world tasks and activities.

The participant was notified that she would receive information about the real world while immersed in VR in the Cue condition, and was either provided with a tutorial about how to request the information (Pull Temporary Condition) or when and how long the information would appear (Push Temporary and Push Permanent). After each round, the participant answered questions about her situation awareness, VR presence, and workload. The participant was also asked to answer three question about the VR tour to check for attentiveness. At the end of the third round, the participant completed a survey that included open questions about what she liked and disliked about the cues and demographics. Demographics questions included age, gender, culture of origin, their Big Five Factors of personality (Rammstedt and John, 2007), previous use of a VR device, and frequency of VR use.

4.8 Measures

4.8.1 Presence in VR

The Spatial Presence Experience Scale (SPES) Hartmann et al. (2016) was used to measure participants’ presence in VR. This scale measures spatial presence across two dimensions: self-location and possible actions. The two dimensions form an 8-item scale. Only the 4 items associated with the Self-Location sub-scale were used in the study since the VR task was a passive task and did not require any actions from the participants.

4.8.2 Perceived workload

NASA-TLX questionnaire Hart and Staveland (1988) was used to measure participants’ perceived workload. This questionnaire incorporates six sub-scales to derive an overall workload score. These sub-scales include mental demand, physical demand, temporal demand, performance, effort, and frustration.

To assess key dimensions of wellbeing relevant to VR experiences, this study focused on positive emotions, motivation and engagement, and mindfulness, as identified by the positive computing framework discussed in Section 2.3. These dimensions capture important aspects of users’ mental and emotional states during VR exposure and provide a comprehensive understanding of how VR impacts wellbeing. Accordingly, validated questionnaires targeting each dimension were selected to measure these constructs in participants.

4.8.3 Positive emotions

The Positive and Negative Affect Schedule (PANAS) Watson et al. (1988) was used to measure participants’ positive emotions. This scale uses a 5-point Likert format to assess positive and negative affect related to a specific experience. Only the positive emotions sub-scale was used in this study.

4.8.4 Motivation and engagement

The Flow Short Scale (FSS) Engeser and Rheinberg (2008) was used to measure participants’ motivation and engagement through their experience of flow—the balance between perceived challenges and personal skills. This scale consists of multiple items rated on a 5-point Likert scale.

4.8.5 Mindfulness

While individual users’ mindfulness skills and characteristics may influence their VR experience (Kiken et al., 2015), mindfulness was viewed as an experienced state in VR to develop VR applications that promote and support this state. Mindfulness was assessed as an experienced state during VR using the mind-subscale of the State Mindfulness Scale (SMS), a 5-point Likert scale questionnaire particularly suited to measuring mindfulness in recent experiences (Tanay and Bernstein, 2013).

4.8.6 Situation awareness

The Situation Awareness Rating Technique (SART) Taylor (2017) is a self-rating measure that was administered after each of the rounds to assess participants’ situation awareness. While objective techniques such as the Situation Awareness Global Assessment Technique (SAGAT) provide direct assessment through freeze-probe queries, they require interrupting the simulation at predefined intervals (Durso et al., 1998; Endsley, 1988). Given that maintaining a continuous and uninterrupted experience was important in this study, the interruptions required by SAGAT could disrupt immersion and introduce confounds in measures of presence and user experience. Therefore, SART was selected as a more appropriate measure for capturing participants’ perceived situation awareness without compromising ecological validity in an immersive VR context.

Participants completed this questionnaire after the No-Cue and Cue conditions. They were not asked about their situation awareness about the real world home environment in the VR Only condition since they were told to only focus on VR.

4.8.7 Fixations

We used fixations to evaluate participants’ visual behavior during the VR task in the Cue condition, since visual information is mainly acquired through fixations (Hoffman, 2016). We extracted fixations with a minimum duration 100 ms (Irwin and Zelinsky, 2002; Lander et al., 2018). The fixation areas of interest were the cues and the VR tour. We explored fixation in terms of fixation rate (fixation count per minute) and duration on the areas of interest.

5 Study 1: results

5.1 Data analysis

The responses to the attention check questions showed that participants were all attentive during the passive task and paid attention to the VR tours. Therefore, the data from all participants were included in the analysis. All the analysis was conducted in SPSS 29.0.1.

A series of split-plot ANOVAs assessed the statistical significance of Information Need and Cue Type, and their potential interaction on participants’ VR presence, perceived workload, and the three wellbeing sub-scales. We used Bonferroni correction to control type I error, and the corrected alpha was 0.01 for these 5 split plot tests.

5.2 Presence in VR

VR presence scores were calculated by averaging the participant’s responses to the SPES after each round. Figure 4 shows the pattern of results for VR presence. Cue Type did not have a significant effect on VR presence scores, F(2,69) = 0.259, p = .773. There was no effect of Information Need on VR presence, F(2, 138) = 2.281, p = .106 There was no interaction effect of Cue Type and Information Need on participant SPES scores, F(4,138) = 0.510, p = .729.

5.3 Perceived workload

The overall perceived workload scores were calculated by averaging the scores of the six subcategories in the NASA-TLX questionnaire. Figure 5 shows the pattern of results for perceived workload. The split-plot ANOVA indicated a statistically significant effect of Information Need on perceived workload, F(2, 138) = 5.793, p = .004, partial eta-squared = 0.077. Cue Type did not have a significant effect on perceived workload scores, F(2,69) = 0.576, p = .565. There was no interaction effect of Information Need and Cue Type on participant workload scores, F(4,138) = 0.916, p = .457.

We conducted paired t-tests to further investigate the effect of Information Need on perceived workload (alpha = 0.0167). The VR Only condition had significantly lower workload scores than the No-Cue condition, t(71) = −2.742, p = .004. The workload scores in the VR Only condition were also significantly lower than the Cue condition, t(71) = −3.351, p < .001. There was no significant difference between the No-Cue and Cue workload scores, t(71) = −0.372, p = .355.

5.4 Positive emotions

The Positive Emotions scores were calculated based on the guidelines in Watson et al. (1988). Figure 6 shows the pattern of results for positive emotions scores. Cue Type did not have a significant effect on Positive Emotions score, F(2,69) = 1.001, p = .372. The sphericity assumption was violated (p < .001). While the F-statistic is considered relatively robust to such violations when group sizes are equal and the design is balanced (Tabachnick et al., 2007), this violation suggests that the results should be interpreted with caution. To further mitigate this, we employed conservative multiple comparison procedures for the main effects of Information Need and the interaction effect (since they both involve the within-subjects factor influenced by sphericity), ensuring that the reported differences remain statistically reliable despite the sphericity violation. Results should be interpreted with caution anyways.

To investigate the main effect of Information Need, we conducted three paired t-tests with alpha = 0.0167. Positive Emotions scores at VR Only condition were significantly higher than that of No-Cue condition, t(71) = 2.503, p = .007. The scores in the No-Cue condition were not significantly different from the Cue condition, t(71) = −1.914, p = .03. There was also no significant difference between positive emotions in VR Only and Cue conditions, t(71) = 1.093, p = .139.

To investigate the effect of the interaction between Information Need and Cue Type, we ran simple main effects twice; one to investigate the effect of Cue Type at each level of Information Need, and one to study the effect of Information Need at each level of Cue Type. None of the three one-way ANOVAs for the effect of Cue Type at each level of Information Need was significant at the alpha = .0167 level. For studying the effect of Information Need at each level of Cue Type, we ran three series of paired t-tests at each level of Cue Type (alpha = .006). There was no significant effect of Information Need at the Push Temporary and Pull Temporary conditions. In the Push Persistent condition, the positive emotions score at VR Only condition was significantly higher than in No-Cue condition, t(23) = 4.336, p < .001. The positive emotions scores at Cue condition was also significantly higher than the No-Cue condition in the Push Persistent group, t(23) = 3.192, p = .002. In this group, there was no significant difference between the VR Only and Cue condition scores, t(23) = 1.540, p = .069.

5.5 Motivation

The Motivation scores were calculated based on the guidelines in Engeser and Rheinberg (2008). Figure 7 shows the pattern of results for motivation scores. Cue Type did not have a significant effect on motivation scores, F(2,69) = 1.909, p = .156. There was no effect of Information Need on VR presence, F(2, 138) = 0.248, p = .781 There was no interaction effect of Cue Type and Information Need on participant motivation scores, F(4,138) = 2.426, p = .051.

5.6 Mindfulness

The Mindfulness scores were calculated based on the guidelines in Kiken et al. (2015). Figure 8 shows the pattern of results for mindfulness scores. Cue Type did not have a significant effect on Mindfulness score, F(2,69) = 0.281, p = .756. The sphericity assumption was violated (p < .001). Although ANOVA is generally considered robust to such violations in balanced designs (Tabachnick et al., 2007), this violation suggests that the results should be interpreted with caution. To maintain statistical rigor, we conducted multiple comparison procedures to investigate the main effects of Information Need and the interaction effect of Information Need and Cue Type (since they both involve the within-subjects factor influenced by sphericity).

To investigate the main effect of Information Need, we conducted three paired t-tests with alpha = .0167. There was no significant difference between the mindfulness scores at the three levels of Information Need (p > .05).

To investigate the effect of the interaction between Information Need and Cue Type, we ran simple main effects twice; one to investigate the effect of Cue Type at each level of Information Need, and one to study the effect of Information Need at each level of Cue Type. None of the three one-way ANOVAs for the effect of Cue Type at each level of Information Need was significant at the alpha = .0167 level. For studying the effect of Information Need at each level of Cue Type, we ran three series of paired t-tests at each level of Cue Type (alpha = .006), none of which was significant.

5.7 Situation awareness

Participants completed the SART questionnaire only in the No-Cue and Cue conditions. The SART scores were calculated following the guidelines described in Taylor (2017). Figure 9 shows the pattern of results for positive emotions scores. We conducted a split plot ANOVA to study the effect of Information Need (2 levels) and Cue Type (3 levels) on participants’ perceived situation awareness. Cue Type did not have a significant effect on SART score, F(2,69) = 0.022, p = .978. There was a significant main effect of Information Need on SART scores, F(2, 138) = 31.519, p < .001, partial eta-squared = 0.314. The interaction effect of Information Need and Cue Type was also significant, F(4, 138) = 3.701, p = .030.

Followup simple main effects analysis was used to further investigate the interaction effect. First, two one-way ANOVAs were conducted to study the effect of Cue Type at each level of Information Need (alpha = 0.025), none of which were significant (p > .05). Paired t-tests were then used to compare SART scores between the two Information Need levels at each level of Cue Type (alpha = .0167). SART scored were significantly higher in the Cue condition than in No-Cue condition in the Push Temporary (t(23) = 2.593, p = .008) and Pull Temporary (t(23) = 6.030, p < .001). There was no significant difference between the SART scores in the Push Persistent condition, t(23) = 1.541, p = .068.

5.8 Fixation

Data preprocessing included removal of blinks and tracking loss periods. Areas of interest (AOIs) were defined as (1) the SA cue region and (2) the primary VR content. Fixation metrics included fixation count per minute and average fixation duration within each AOI.

Figure 10 shows the pattern of results for participants’ average number of fixations. A one-way ANOVA showed a significant effect of Cue Type on the total number of fixation on cues, F(2, 69) = 58.618, p < .001, partial eta-squared = 0.630. Followup Tukey test showed that participants in the Push Persistent condition fixated on the cues significantly more times than the participants in each of the other two cue types.

Figure 11 shows the pattern of results for participants’ average duration of fixations. There was also a significant effect of Cue Type on the fixation duration on cues, F(2, 69) = 3.195, p = .047, partial eta-squared = 0.085. Followup Tukey test showed that participants in the Push Persistent condition fixated on the cues significantly longer than the participants in the Pull Temporary condition, p = .039.

5.9 Post-experiment survey

In addition to the quantitative data, qualitative data from the open questions after the experiment provided insight into the participants’ experience of using a VR device in a home environment. The open questions asked participants to discuss what they liked and disliked about receiving cues. Several rounds of open coding were conducted. Generated codes were shared and discussed by the research team after each iteration of coding. The first round of coding closely followed the text. The next round of coding was more high-level and resulted in codes such as “permanency,” “distraction,” “autonomy,” and “information and details” which are discussed in this section.

When asked what they disliked about receiving cues about the real-world home environment, almost all participants mentioned being distracted by them. On the other hand, many participants expressed their appreciation for the cues as they provided them with information about the home environment. For instance, some participants liked that the headset “provided more information to what I was hearing from the real world” (P5) and that it was “… informative, tells me exactly what happened so I don’t have to worry about what happened” (P7). Moreover, a few participants liked the cues since they received information without the need to take the headset off (P59, P60, P61). Further, some participants mentioned that the cues allowed them to assess the social situation around them and decide what action to take in response to the real-world events. For instance, P40 said, “[I liked] knowing who was coming into the room. Since it was just my roommate, I didn’t mind, but if this was real and my partner walked in, I would take off the headset pretty quickly.”

Some of the participants in the Push Persistent condition liked the permanency of the cues since they could “look back later to remember what had happened” (P32) and did not have to “memorize what exactly happened because I could remind myself by looking at the screen” (P61). On the other hand, a few participants disliked the permanency of the cues and “would have preferred if they disappeared” (P26). For instance, P93 mentioned that he “disliked that it took my focus away from the information and views of Las Vegas. I couldn’t listen as well because my brain kept reading the pop-up cues.”

Participants in the Pull Temporary condition stated that they liked having the autonomy to turn the cues on and off. Interestingly, a few participants in the Pull Temporary and Push Temporary conditions mentioned that they wished they had the option to learn more about the real-world events. For instance, P59 in the Pull Temporary condition expressed that he wished “the pop-ups build on each other … so I can always refer to everything that has happened while I was in VR rather than what happened in the moment.”

6 Study 1: discussion

Providing information about the real-world environment while users are immersed in VR could be both beneficial and distracting for them (Fereydooni et al., 2022; Li et al., 2023; Schleußinger et al., 2023). There are many features of these real-world cues that could be tweaked to mitigate the distracting effect of the cues on users’ VR experience when using VR in a home setting. This study investigated how this issue may be mitigated perceptually, by studying users’ autonomy and cue persistence in the virtual environment.

6.1 Effect of information need

RQ1.1. How does users’ need to attend to real-world tasks while immersed in VR influence their experience?

Answering the first research question, the observed significant decrease in participants’ perceived workload in the VR Only condition, compared to when they had to attend to real-world home events, suggests a notable influence of users’ need to engage in real-world tasks on their immersive VR experience. This is understandable since they did not have to allocate cognitive resources to the home environment events in the VR Only condition. However, the effect size for this finding was relatively modest, indicating that while the effect is statistically reliable, its practical impact may be limited. For VR developers, this suggests that the background cognitive load of simply needing to know about the real world is a stable factor that does not necessarily break the user experience. Instead of attempting to eliminate this distraction entirely, design efforts should be directed toward optimizing the Information Architecture of the cues.

RQ1.2. How does providing cues about the home environment in the virtual environment affect users’ experience?

Addressing the second research question, the results indicate that providing cues about the home environment in the virtual environment did not lead to a significant difference in participants’ perceived workload in comparison to the No-Cue condition. This finding suggests that the provision of cues did not impose an additional cognitive burden on participants. In this study, participants in the No-Cue and Cue condition were asked to keep track of the real-world home events while attending to the VR task, creating a dual-task situation. The multiple resource theory helps explain that this dual-task situation potentially increased participants’ perceived workload by overwhelming them with information from both the real and virtual environments (Basil, 1994). This spike in participants’ perceived workload due to the instructions may have lead to lower power in detecting the effect of the cues. This shows how the allocation of cognitive resources varies based on the specific instructions and task conditions, influencing participants’ perceived workload.

6.2 Effect of cue type

RQ1.3. How does the persistence of cues affect users’ experience?

Regarding the third research question, participants in the Push Persistent condition demonstrated significantly higher positive emotions in the Cue condition than in the No-Cue condition. This suggests that Push Persistent cues made participants feel good and secure about the availability of information. This may be due to the fact that participants in this condition had the option to look back at the cues if they needed to, to learn about and remember what happened in the real-world home environment—an externalized form of memory and information retrieval. The open-ended questions provide corroborating evidence supporting this, as some participants expressed their positive appreciation for the permanency of the Push Persistent cues. The permanency of the Push Persistent cues also potentially led participants to be fixated on the cues at a significantly higher rate (i.e., fixation count). However, these Push Persistent cues did not significantly change participants’ situation awareness, as opposed to Push Temporary and Pull Temporary cues that significantly increased participants’ situation awareness. In other words, even though participants in the Push Persistent condition looked at the cues more often than the participants in the Push Temporary and Pull Temporary conditions, they did not perceive and digest the information to increase their situation awareness. The way participants used the cues could be viewed from the extended cognition lens (Kiverstein, 2018). That framework suggests that cognitive processes are not confined to the individual’s brain but extend into the external environment through interactions with tools and artifacts. Not observing a significant difference between participants’ situation awareness in Cue and No-Cue conditions indicates a nuanced relationship between the extended cognitive tools and the cognitive processes at play for participants in the Push Persistent condition. In other words, while participants were actively using the Push Persistent cues as external tools —indicated by significantly higher fixation count— the cognitive integration and internalization of the information into participants’ situational understanding did not occur to the same extent as with other cue types.

RQ1.4. How does users’ autonomy in receiving cues affect their experience?

Addressing the fourth research question, participants receiving the Pull Temporary cues demonstrated significantly lower fixation duration than those with Push Persistent cues. The participants in the Pull Temporary condition had autonomy in deciding when to retrieve information about the real-world home environment. This autonomy likely led to a more targeted and efficient use of the cues, resulting in lower fixation duration. Since participants had control over when and how they accessed information, they could quickly extract relevant details without prolonged fixation. On the other hand, participants in the Push Persistent condition had cues continuously present on the screen. This constant availability might have led to a more passive interaction, where participants may not have felt the need to extract information immediately, causing longer fixation durations. The permanency of the Push Persistent cues potentially encouraged prolonged observation without the same urgency for information retrieval that autonomy in the Pull Temporary condition afforded.

7 Study 1: conclusion

The results of this study shine some light on how some of the perceptual aspects of cues that provide information about the real world in VR influence users’ experience. The lower perceived workload when participants had to only focus on VR confirms the impact of users’ real-world engagement on their immersive experiences. Interestingly, while cues did not significantly affect perceived workload between the conditions in which participants had to attend to the real-world, they elicited positive emotions in participants exposed to Push Persistent cues. Furthermore, autonomy in cue retrieval appears to influence the efficiency of information processing, reflecting the nuanced interplay between the cognitive processes of participants and the external cues provided.

These findings highlight the complexity of integrating cues into VR experiences and suggest avenues for future research. Understanding the interplay between cues and user cognition is crucial for enhancing VR integration into diverse real-world settings, ultimately ensuring user comfort, safety, and engagement.

8 Study 2: overview

Study 1 investigated how the distracting nature of the cues may be mitigated perceptually, by studying users’ autonomy and cue persistence in the virtual environment. As a next step in the investigation, Study 2 explored the information architecture of the cues and the effect of delivering the information through ambient displays.

Ambient displays are designed to subtly and unobtrusively convey information to users, seamlessly integrating into the environment without requiring direct attention (Matthews et al., 2003). Building on this concept, Pousman and Stasko (2006) introduced four design patterns for ambient information systems. These include the Symbolic Sculptural Display, Multiple-Information Consolidators, Information Monitor Display, and High Throughput Textual Display. Symbolic Sculptural Displays employ symbolic or abstract physical forms to represent information. These abstract sculptural displays are visually appealing and show very few pieces of information, usually a single element. Multiple-Information Consolidators aggregate multiple pieces of abstracted information into a single display. Information Monitor Displays communicate multiple sources of information through visual metaphors, and can afford different interactions and design choices. These systems may be aesthetic, but their primary function extends beyond mere visual appeal. High Throughput Textual Display systems are those that use text and very simple graphics (i.e., icons) to communicate information.

The effect of the type of displays, however, may depend on the task type performed in VR; in other words, it is not clear if users’ awareness needs are universal or application-specific (O’Hagan et al., 2022). Moreover, the existing literature suggests that active and passive VR tasks have different implications for users’ memory and task impression (Hine and Tasaki, 2019). Hence, this study investigated the effect of different information architectures of cues in various VR task types—particularly the activity level required in the task—through the following research questions:

RQ2.1. How do different information displays in VR influence users’ experience, in terms of their perceived workload, and perceived situation awareness about the real world, VR presence, and physiological responses, specifically gaze behavior?

RQ2.2. How does the activity level of the tasks that users engage with in VR influence the effect of the information displays?

9 Study 2: methods

9.1 Participants

Study 2 included 52 participants. Table 1 provides a breakdown of participants’ demographics in each of the conditions. The age of the participants (29 female, 23 male) ranged from 18 to 23 years (M = 19.7, SD = 1.26). Participants were recruited from the university’s undergraduate research subject pool. Participation criteria included sufficient mobility to use a VR device, and normal or corrected-to-normal vision and hearing. Among all participants, 9.6% owned a VR headset, and 26.9% stated that they use a VR device at least a few times per year.

Table 1. Demographics of participants in each of the conditions in Study 2.

9.2 Ethics statement

This study was reviewed and approved by the University Institutional Review Board (IRB). Participants were compensated with one hour of coursework extra credit for their participation in the hour-long study session. To guarantee completely voluntary participation and eliminate potential coercion, an equitable alternative non-research assignment was made available for students to earn the exact same credit amount within their respective coursework.

9.3 Experiment design

Study 2 was a 2 × 2 mixed design, studying the effect of two independent variables (Display Type, and VR Activity Level) on users’ perceived situation awareness, perceived workload, VR presence, and wellbeing. Display Type was a within-subject variable derived with two levels (Text and Embedded). VR Activity Level was a between-subjects variable with two levels (Active and Passive). Each participant was randomly assigned to a VR Activity Level condition and performed the experimental tasks in two rounds to experience the two levels of Display Type. The order of the conditions was counterbalanced using a Latin Square Design.

9.4 VR experience

The VR experience in the Passive task consisted of sitting in a virtual drive-in theater and watching a short movie. The order of the movies was fixed; “Standby” Regan (2017) was shown during the first round, and “Cropped” Thomas (2025) during the second round. Each movie was about 6 min long to minimize the buildup of cyber-sickness symptoms and fatigue (Davis et al., 2015; Kennedy et al., 2000). The VR experience in the Active task consisted of playing a game located in a virtual barn. Fruit were falling from the sky and each participant had to catch them using a bucket. The bucket could move horizontally, and was controlled by the trigger buttons of the headset controllers, such that the left controller moved the bucket to the left and the right controller to the right. Occasionally, mud-balls would also fall from the sky, which the participant had to avoid collecting. Similar to the Passive task, each round of the Active task was about 6 minutes.

9.5 Display type

Five items of information were selected to be displayed: traffic on a local highway, number of other people in the apartment, number of text messages, number of emails, and the time left on the oven to finish baking a cake. These items are examples of information people typically seek to maintain awareness of when using a VR headset (Li et al., 2020). The two levels of Display Type were derived and designed to communicate the status of these items based on the framework developed by Pousman and Stasko (2006).

Pousman and Stasko’s framework consists of four types of ambient displays. While Symbolic Sculptural Displays offer visually engaging representations, they were not suitable for this study’s objective of conveying multiple pieces of information. Additionally, Information Monitor Displays offer diverse methods for alerting users to changes in the source data, ranging from subtle cues to more intrusive interruptions. Given the wide range of design choices within this category, we opted to exclude it from this study.

Text Cues were based on High Throughput Textual Displays in the framework, and were designed similar to the cues in Study 1. These cues were positioned on a board at the lower middle of the user’s field of view to provide information without blocking the participant’s view of the video or the game. Figures 13, 15 demonstrate the Text Cues in the Passive and Active tasks, respectively. Embedded Cues were based on Multiple Information Consolidators in the framework; these cues are abstract pictorial representation of the information that deliver the information in a peripheral manner (Miller and Stasko, 2002). Table 2 shows the individual objects in the virtual environment that represented the information about the real-world home environment. Figures 12 and 14 demonstrate these Embedded Cues in the Passive and Active tasks, respectively.

Table 2. Pictorial representations of the real-world events in the virtual environment for active and passive tasks.

9.6 Procedure

This study was conducted at the same research facility as Study 1. After the participant signed the informed consent form, she was screened for simulator sickness (Gable and Walker, 2013). The VR experience during the screening protocol consisted of watching a short movie in the drive-in theater for the Passive Group and practicing the game for the Active group. After passing the screening, the participant was calibrated for the eye tracking system. She was then provided with the real-world scenario that she had to imagine for the study: she was asked to imagine that she lives in the apartment with two roommates, and that she is waiting for her teammates to arrive at her apartment to work on a class project. She was asked to imagine that she is baking a cake for her teammates and keeping track of the time that the cake is in the oven. The participant was also told that she may receive texts and emails on her phone, and she is tracking the traffic status on a local highway to have an idea when her teammates will arrive. Finally, she was asked to imagine that she decided to use a VR headset before her teammates arrive. The participant was informed that none of the real-world home events are urgent and no action is required in response to them. This scenario was designed so that all participants were in a consistent at-home frame of mind. The participant was also asked to keep track of these tasks while using VR. This requirement was in place to simulate users’ need for keeping track of real-world tasks and activities.

The participant was notified that she would receive information about the real world while immersed in VR, and was shown what each notification would look like. After each round, the participant answered questions about her situation awareness, VR presence, and workload. In the Passive group, the participant was also asked to answer three questions about the short movie to check for attentiveness. At the end of the second round, the participant completed a survey that included open questions about what she liked and disliked about the cues and demographics. Demographics questions included age, gender, culture of origin, their Big Five Factors of personality (Rammstedt and John, 2007), previous use of a VR device, and frequency of VR use.

9.7 Measures

The measures used in Study 2 were the same as the ones used in study 1: presence in VR, perceived workload, positive emotions, engagement, mindfulness, situation awareness, fixation duration, and fixation count.

10 Study 2: results

10.1 Data analysis

The responses to the attention check questions showed that participants were all attentive during the passive task and paid attention to the VR tours. Therefore, the data from all participants were included in the analysis. All the analysis was conducted in SPSS 29.0.1, R (v4.3.0), and RStudio (v2023.09.0).

A series of split-plot ANOVAs assessed the statistical significance of Display Type and VR Activity Level, and their potential interaction on participants’ VR presence, perceived workload, and the three wellbeing sub-scales.

10.2 Presence in VR

Figure 16 shows the pattern of results for perceived workload. Activity Level did not have a significant effect on VR presence scores, F(1, 50) = 0.080, p = .779. Display Type did not have a significant effect on perceived workload scores, F(1,50) = 0.356, p = .553. There was no interaction effect of Activity Level and Display Type on participant workload scores, F(1,50) = 0.356, p = .553.

10.3 Perceived workload

Figure 17 shows the pattern of results for perceived workload. The split-plot ANOVA indicated a statistically significant effect of Activity Level on perceived workload, F(1, 50) = 8.196, p = .006, partial eta-squared = 0.141, such that participants in the Active condition reported higher perceived workload. Display Type did not have a significant effect on perceived workload scores, F(1,50) = 0.913, p = .344. There was no interaction effect of Activity Level and Display Type on participant workload scores, F(1,50) = 0.646, p = .425.

10.4 Wellbeing: motivation

Figure 18 shows the pattern of results for motivation score. Activity Level did not have a significant effect on motivation scores, F(1, 50) = 0.070, p = .793. Display Type did not have a significant effect on motivation scores, F(1,50) < 0.001, p = .999. There was no interaction effect of Activity Level and Display Type on motivation scores, F(1,50) = 0.323, p = .573.

10.5 Wellbeing: mindfulness

Figure 19 shows the pattern of results for mindfulness score. Activity Level did not have a significant effect on mindfulness scores, F(1, 50) = 0.126, p = .724. Display Type did not have a significant effect on mindfulness scores, F(1,50) = 0.018, p = .895. There was no interaction effect of Activity Level and Display Type on participant mindfulness scores, F(1,50) = 0.441, p = .510.

10.6 Wellbeing: positive emotions and situation awareness

Situation awareness scores and positive emotions scores were correlated (r = .33, p = .016). Hence, a MANOVA analysis was used to investigate the effect of Activity Level and Display Type on them simultaneously.

The multivariate normality assumption was not rejected at the alpha = 0.05 level using the Mardia test; skew = 1.257 (p = .868), kurtosis = −0.087 (p = .929). The homogeneity of covariance dispersion also did not achieve significance, supporting the assumption of equal population variance-covariance matrices, Box M = 16.957, p = .115.

10.6.1 Main effect of activity level

To investigate the main effect of Activity Level on positive emotions and situation awareness, these scores were averaged across Display Type. The scatter plot of average positive emotions and average situation awareness scores in Figure 20 visually depicts no substantial difference between the conditions in the main effect of Activity Level. MANOVA analysis also did not show a significant main effect of Activity Level on the combination of dependent variables of positive emotions and situation awareness, V = .011, F(1,50) = 0.283, p = .755. As the main effect of Activity Level was not significant, no canonical correlations were provided and no post-hoc analyses were performed.

10.6.2 Main effect of display type

To investigate the main effect of Display Type on positive emotions and situation awareness, the difference scores between Embedded and Text cues were calculated. A subsequent MANOVA test was conducted on the positive emotions and situation awareness difference scores. The main effect of Display Type was derived from the intercept in the MANOVA analysis. In the MANOVA test, a significant main effect of Display Type was found on the multivariate dimension of positive emotions and situation awareness, V = .175, F(1, 50) = 5.21, p = .008. Figures 21, 22 illustrate the effect of Display Types on situation awareness and positive emotions scores. The significant dimension, or canonical variate, had different correlations with the dependent measures. The situation awareness difference scores correlated with the significant dimension in a strong, positive manner, r = .81, while positive emotions difference scores correlated in a positive, but weaker manner, r = .57. The difference in canonical correlation sizes suggests that the situation awareness scores are driving the significance of the main effect in the MANOVA more than positive emotions scores. Additionally, the same directionality of correlation and the standardized discriminant coefficients suggests that the Embedded Cues resulted in higher situation awareness and positive emotions than the Text Cues.

10.6.3 Interaction effect of activity level and display type

The interaction effect of Activity Level and Display Type was derived from the main effect of the MANOVA discussed in Section 10.6.2. Figure 23 visually depicts the difference scores for the dependent variables (Embedded minus Text) divided by Activity Level. However, the MANOVA test did not show a significant interaction effect of Activity Level and Positive Emotions on the combination of dependent variables of positive emotions and situation awareness, V = .005, F(1,50) = .136, p = .873. As the interaction effect was not significant, no canonical correlations were provided and no post-hoc analyses were performed.

10.7 Fixation count and fixation duration

Fixation count and duration values were correlated (r = .42, p = .002). Hence, a MANOVA analysis was used to investigate the effect of Activity Level and Display Type on them simultaneously.

The multivariate normality assumption was rejected at the alpha = 0.05 level using the Mardia test; skew = 43.724 (p < 0.001), kurtosis = 6.057 (p < .001). The assumption of homogeneity of covariance dispersion was not rejected, Box M = 108.777, p = .115.

Given the violation of multivariate normality, Pillai’s Trace was used for multivariate testing, as it is considered the most robust statistic to departures from normality and covariance assumptions (Tabachnick et al., 2007). Additionally, the balanced design and relatively similar log determinants of the covariance matrices across conditions (Active = 6.337, Passive = 9.041) further support the robustness of the analysis.

10.7.1 Main effect of activity level

To investigate the main effect of Activity Level on fixation count and duration, these scores were averaged across Display Type. A subsequent MANOVA test was conducted on these averaged scores. A significant main effect of Activity Level was found on the multivariate dimension of fixation duration and fixation count, V = .541, F(1, 50) = 28.85, p < .001. Figure 24 illustrates the main effect between the Active and Passive tasks. The significant dimension, or canonical variate, had different correlations with the dependent measures. The average fixation duration correlated with the significant dimension in a strong, positive manner, r = .75, while average fixation counts correlated in a positive, but weaker manner, r = .55. The difference in canonical correlation sizes suggests that the fixation duration values are driving the significance of the main effect in the MANOVA more than fixation count values. Additionally, the same directionality of correlation and the standardized discriminant coefficients suggests that the Passive Task resulted in higher fixation duration and higher fixation counts than the Active Task.

10.7.2 Main effect of display type

To investigate the main effect of Display Type on fixation duration and fixation count, the difference scores between Embedded and Text cues were calculated. A subsequent MANOVA test was conducted on the fixation duration and fixation count difference values. The main effect of Display Type was derived from the intercept in the MANOVA analysis. In the MANOVA test, a significant main effect of Display Type was found on the multivariate dimension of fixation duration and fixation count, V = .783, F(1, 50) = 88.64, p < .001. Figures 25, 26 illustrate the effect of Display Types on fixation duration and fixation count values. The significant dimension, or canonical variate, had different correlations with the dependent measures. Fixation duration difference scores correlated with the significant dimension in a strong, positive manner, r = .87, while fixation count difference scores correlated in a weaker negative manner, r = −.60. The difference in canonical correlation sizes suggests that the fixation duration values are driving the significance of the main effect in the MANOVA more than fixation count values. Additionally, the difference in the directionality of correlation and the standardized discriminant coefficients suggests that the Text Cues resulted in higher fixation duration but lower fixation counts than the Embedded Cues.

10.7.3 Interaction effect of activity level and display type

The interaction effect of Activity Level and Display Type was derived from the main effect of the MANOVA discussed in Section 10.7.2. A significant interaction effect of Display Type was found on the multivariate dimension of fixation duration and fixation count, V = .596, F(1, 50) = 36.20, p < .001. The scatter plot of average positive emotions and average situation awareness scores in Figure 27 visually depicts the difference scores for the dependent variables (Embedded minus Text) divided by Activity Level. The canonical variate had different correlations with the dependent measures. Fixation count difference scores correlated with the significant dimension in a strong, positive manner, r = .90, while fixation duration difference scores correlated in a weaker negative manner, r=−.55.

This significant interaction effect signifies that the difference between Embedded and Text Cues depends on the Activity Level. Figure 28 depicts boxplot distributions of fixation count values for Display Types and Activity Levels. This plot indicates that the number of times participants looked at Text Cues was comparable between Active and Passive tasks. However, participants looked at Embedded Cues more often in the Passive Task than in the Active Task.

Figure 29 depicts boxplot distributions of fixation duration values for Display Types and Activity Levels. This plot indicates that participants looked at Text cues longer in the Passive Task than those in the Active Task. Participants also looked longer at Embedded Cues in the Passive Task than those in the Active Task, but to a lesser extent than Text Cues.

10.8 Post-experiment survey

In addition to the quantitative data, qualitative data from the open questions after the experiment provided insight into the participants’ experience of using a VR device in home environment. The open questions asked participants to discuss what they liked and disliked about each of the Display Types. Several rounds of open coding were conducted. Generated codes were shared and discussed by the research team after each iteration of coding. The first round of coding closely followed the text. The next round of coding was more high-level and resulted in codes which are discussed in this section.

One of the themes in participants’ responses about what they disliked about Embedded Cues—in both Active and Passive Tasks—was remembering the coding of the cues. For instance, P28 mentioned, “I disliked this because I had to think about which object represented what and had a lot more to memorize,” when discussing what he disliked about the Embedded Cues. Moreover, participants did not like having to decode every change in the virtual environment back to their real-world meaning; “The embedded cues just felt natural inside the environment and just a glance let me know the information. But parsing the information into real world information took a bit of effort to keep track of” (P29). However, many participants explained that if they remembered the coding, the information retrieved from Embedded Cues were more memorable than what was learned from Text Cues; “I had to think about the embedded cues so I remembered them better” (P10). Additionally, P55 stated, “it is more challenging to remember what each object symbolized… I did like, however, that if I really paid attention and could remember the symbolizations, each object was clearly separated from the rest and easier to remember.” Distinction of visual representations in Embedded Cues (e.g., birds representing texts vs. fence color representing traffic) was another factor that made the information from these cues more memorable than Text Cues (P57). On the other hand, for Text Cues, it was “easy to gain new information, but it wasn’t easy to remember after reading once” (P40). Interestingly, to remember what was happening in the real world, P49 “came up with abbreviations such as ‘200M’ meaning 2 people, 0 texts, 0 emails, and medium traffic,” but mentioned that with Embedded Cues there was “no need for that as the cues did more than enough to help me remember.”

Another difference between Embedded and Text Cues that created a theme in participants’ responses was the location of the Cues. For instance, P55 explained, “Because the text cues were all on top of one another, it was also somewhat difficult to remember them as well as opposed to embedded cues with more separate boundaries.” This participant is referring to the fact that Embedded Cues were more dispersed in the virtual environment, whereas Text Cues were all written on a board in front of her. Although the location of Text Cues was not as favorable in terms of remembering the information, many participants—particularly in the Active Task—mentioned preferring text cues as them being in the same area allowed participants to focus on the VR game more (e.g., P1, P7, P9, P14). For example, P7 stated, “Since all the visual information was in one place, I was able to plan my moves for the game better, since I could glance at the text board and use my peripheral vision to play the game,” when referring to Text Cues. Similarly, participants in the Active Task mentioned that the location of Embedded Cues and having to “track visual cues across the screen” led these cues to be “significantly more distracting than the first run [Text Cues].” Many other participants discussed this additional divided attention caused by Embedded Cues (e.g., P9, P12, P14, P22, P43).

Almost all participants in the Passive Task mentioned the difficulty to track changes with Text Cues; some mentioned that “it was not very noticeable when there’s change” (P14), and that they were not “as tuned into the changes on the board” (P7). Some participants mentioned that they did not like not knowing exactly when something happened in the real world (P30, P32). In contrast, these participants expressed their liking for Embedded Cues since the changes were more noticeable.

The last factor that separated Embedded and Text Cues for participants was the speed of information extraction: participants mentioned that they could glance at Embedded Cues (e.g., P4, P27, P29), whereas reading the information in the Text Cues took their attention away from the VR task for longer (P36).

11 Study 2: discussion

The findings from Study 2 highlight the dynamic interplay between the information architecture of cue, task type, and users’ cognitive processes in shaping their interaction with information displays in VR.

RQ2.1. How do different information displays in VR influence users’ experience?

Answering the first research question in Study 2, participants’ higher perceived situation awareness with Embedded Cues compared to Text Cues—regardless of Activity Level—suggests that the design of cues can impact users’ sense of awareness about the real-world home environment. Endsley suggested that attention and memory influence the accuracy and completeness of situation awareness that a person derives from the environment (Endsley and Garland, 2000). The qualitative data demonstrates how each of these two factors may have contributed to participants’ higher perceived situation awareness with Embedded Cues than with Text Cues.

First, how people direct their attention has a fundamental impact on which portions of the environment are incorporated into their SA (Endsley and Garland, 2000). In this light, Endsley argues that the degree to which cues draw attention is important. Participants mentioned that the Embedded Cues captured their attention more effectively than the Text Cues, and their fixation count was higher for Embedded Cues. Hence, whereas the distinct coding of the cues and their dispersed location was not favorable for the participants, the Embedded Cues directed participants’ attention more, leading to higher perceived situation awareness.

In terms of the effect of memory on participants’ situation awareness, Jones and Endsley found that working memory losses (where information was initially perceived and then forgotten) explained some situation awareness shortcomings (Jones and Endsley, 1996). This effect was seen in the qualitative data, where participants mentioned forgetting the information from Text Cues after reading them, but reported a better retaining of the information from Embedded Cues.

From a practical design perspective, these findings suggest that while Text Cues are a viable baseline for awareness, they rely heavily on active working memory, which can lead to information decay. In contrast, Embedded Cues capitalize on “glanceability”—the ability to extract information through pre-attentive visual processing. This suggests that for high-immersion VR applications where cognitive load is high, designers should prioritize embedded iconography over textual boards to facilitate more durable situation awareness with lower cognitive effort.

RQ2.2 How does the activity level of the tasks that users engage with in VR influence the effect of the information displays?

Participants’ perceived workload was higher in the Active Task than in the Passive Task. This was expected since the Active Task involved engaging in a game where participants had to actively catch some falling objects while avoiding others, requiring continuous attention and physical interaction. In contrast, the Passive Task simply involved watching a short movie without any active involvement or decision-making on the part of the participants.

One of the differences between Text and Embedded Cues was how long participants looked at them (i.e., fixation duration); participants looked at Text Cues longer than Embedded Cues. This is understandable since they had to read the information from Text, whereas information was acquired by a glance from Embedded Cues, as mentioned in the qualitative data. In other words, reading text required more processing and decoding of information, whereas Embedded Cues were perceived faster—regardless of them being decoded. This aligns with existing literature suggesting that images are perceived more quickly than text (Potter et al., 2014; Tuscher, 2022). Moreover, information in a textual form may not be comprehensible to people in a short glance, but images are easier to grasp than text, especially within a short time limit (Helfman, 1999).

Although participants looked longer at Text Cues than Embedded Cues, the difference between Embedded and Text Cues depended on the Activity Level; participants looked at Text Cues for much longer in the Passive Task. This result is in line with expectations as the Passive Task allowed participants more time to engage in detailed information processing compared to the Active Task. During the Passive Task, participants likely had fewer distractions and could afford to allocate more time to carefully read and process the textual information presented by Text Cues. In contrast, the Active Task required participants to quickly extract relevant information while concurrently attending to the ongoing game, resulting in shorter fixation durations for both Display Types.

This also explains why participants looked more often (i.e., fixation count) at Embedded Cues than at Text Cues, and why this frequency was much higher in the Passive Task than in the Active Task. The faster comprehension of Embedded Cues potentially encouraged participants to repeatedly shift their gaze towards them, seeking to quickly acquire relevant information without the need for looking at them longer. In contrast, looking at Text Cues longer may have resulted in fewer gaze shifts towards them. Similar to fixation duration, the higher frequency of gaze shifts towards Embedded Cues in the Passive Task reflects the fact that participants in this task could afford to look at them more often, whereas in the Active Task they need to focus more on the game, leading to fewer gaze shifts towards both Display Types.

Another interesting observation was how participants perceived the cues differently depending on what Activity Type they were assigned to. Participants in the Active Task discussed liking Text Cues because they were all located in the same area. These participants expressed their dislike for how spread out information in the Embedded Cues were. This can be explained by the limited time they had to divert their attention from the VR game, so they could rely on the consistent location of the Text Cues. In the Passive Task, however, participants noted that they struggled to identify when the information on the Text Cues was updated. These participants liked how noticeable the changes in Embedded Cues were. Different preferences and focusing on different aspects of the cues could be attributed to the task types and distinct needs of participants in each one of them; the Active Task required fast access to information without disrupting ongoing activities, while the Passive Task required clear and noticeable cues to facilitate information processing during periods of lower cognitive engagement. Thus, participants’ preferences for display types were shaped by the specific cognitive and situational demands of their assigned tasks, highlighting the importance of tailoring cue designs to align with users’ varying needs and task contexts. This finding aligns with the principles of threaded cognition (Salvucci and Taatgen, 2008), illustrating how individuals adapt their cognitive strategies to effectively manage multiple concurrent tasks in different contexts.

12 Study 2: conclusion

Study 2 investigated the influence of different information displays with different information architectures on users’ VR experience and the impact of task activity level. Participants exhibited higher perceived situation awareness with Embedded Cues compared to Text Cues, irrespective of Activity Level, emphasizing the role of cue design in shaping users’ awareness of the real-world environment. Attention and memory emerged as critical factors, with Embedded Cues effectively capturing participants’ attention and facilitating faster information processing. These findings emphasize the dynamic interplay between information architecture of cues, task type, and user cognition in VR environments, and their role in shaping participants’ interaction with information displays in VR environments.

13 General discussion: emergent themes

13.1 Levels of situation awareness

SA is a complex construct that plays a crucial role in human performance in a wide range of domains, from aviation to everyday activities like driving and household tasks (Endsley and Garland, 2000). In a model developed by Endsley, SA involves the dynamic process of perceiving, comprehending, and projecting information about the environment to understand what is happening, what it means, and what might happen next (Endsley and Jones, 2013). When an individual uses a VR headset, their most common means of acquiring SA about the outside world—their eyes—is blocked, making a mechanism to provide SA about the real world crucial to their experience.

In Study 1, participants expressed both positive and negative reactions to receiving cues about the home environment. While many appreciated the additional information provided by the cues, such as awareness of social situations and the ability to remember past events, others found them distracting and preferred them to be temporary. This variability in participants’ reactions reflects the different levels of perception and comprehension of the cues, as described by Endsley’s model. The results from Study 2 provide insights into how different information displays in VR influence users’ experience. The distinct effects of Embedded and Text cues on participants’ attention and memory resulted in a greater perceived situation awareness with Embedded Cues compared to Text Cues. This suggests that the design of cues can impact users’ sense of awareness about the real-world environment, highlighting Endsley’s notion that attention and memory influence the accuracy and completeness of situation awareness derived from the environment.

Furthermore, participants commonly found cues to be distracting, albeit to varying degrees depending on the type of cues and the context of the task. While cues were designed to enhance users’ awareness and facilitate task performance, they often resulted in distraction, negatively impacting the overall VR experience. This distraction stemmed from factors such as the need to decode or remember the meaning of cues, the location or dispersal of cues within the virtual environment, and the cognitive load imposed by constantly attending to cues while simultaneously engaging in VR tasks.

These studies, together, shed some light on how changing various features of cues results in them interacting with different levels of situation awareness and constructs affecting it. Therefore, when suggesting a new design for the cues, it is essential to understand which specific levels and constructs of situation awareness they influence. By considering these factors, designers can tailor cues to enhance users’ perception, comprehension, and projection of information in virtual environments effectively. These findings provide initial design guidance derived from controlled experimental conditions, and should be validated in more complex and ecologically realistic VR scenarios.

13.2 Emotion vs. Cognition

Storbeck and Clore (2007) argues that cognition and emotion are deeply interconnected, with each influencing the other in various ways. Drawing on this framework, the relationship between cognition and emotion in the context of SA cues in VR may be analyzed.

In Study 1, participants’ higher positive emotions in response to Push Persistent cues align with Storbeck’s framework, illustrating how cognitive processes (i.e., perception of information availability) can regulate emotional experiences. However, the lack of significant changes in situation awareness despite increased fixation count on Push Persistent cues suggests that emotional responses may not always translate into cognitive enhancements.

Previous work highlights the divergence between cognitive evaluations and emotional responses (Fereydooni et al., 2022). However, in the present two studies, there are instances where cognitive processes, such as perception and attention, modulate emotional experiences and vice versa, demonstrating the bidirectional nature of cognition-emotion interactions. These findings underscore the importance of considering both cognitive and emotional factors in understanding human behavior and decision-making processes, particularly in contexts such as VR use.

13.3 Practical applications

This paper offers insights that can impact the design of next-generation VR headsets as more technology-rich contexts emerge, and as users manage the increase of VR in their work days and leisure. Understanding users’ experiences and needs in VR environments is crucial for designing VR headsets that are user-centric. By focusing on users’ wellbeing, VR headset manufacturers can prioritize features that enhance users’ comfort, mitigate potential risks, and provide a seamless and enjoyable experience.

The exploration of SA cues in VR environments may inform the design of VR headsets with built-in features that enhance users’ awareness of the real world; these features may include features and cameras that detect various activities in the real-world environment. Moreover, these insights can extend to the application level, enabling the seamless integration of cues that are tailored to each specific application, thereby enhancing the overall user experience.

Although significant effects were observed, not all statistical assumptions were fully satisfied. While robust test statistics were used, these violations may affect the precision of the results. Accordingly, findings should be interpreted as indicative rather than definitive, and future work should confirm these effects under conditions that better satisfy statistical assumptions.

14 Limitations

There are several limitations to this study that should be considered when interpreting the findings. First, although the experiment was conducted in a purpose-built facility designed to mimic a suburban home environment, there are inherent differences between this simulated setting and participants’ actual living environments. Despite efforts to create a realistic and relatable scenario, participants may not have experienced the same level of personal relevance, responsibility, or emotional engagement as they would in their own homes. This may have influenced their perception of real-world events and their interaction with the SA cues.

Second, the participant sample consisted primarily of young, technologically proficient undergraduate students. This relatively homogeneous demographic limits the generalizability of the findings to broader populations. Prior research suggests that age and technological familiarity can significantly influence situation awareness processes, cognitive load management, and attitudes toward immersive technologies (Arning and Ziefle, 2009; McGlynn et al., 2018; Simcock et al., 2006). As such, older adults or individuals with lower levels of technological experience may respond differently to SA cues in VR. Future work should examine these effects across more diverse populations to better understand the broader applicability of the findings.

Third, the VR exposure in each condition was relatively brief (approximately 6–7 min). While this duration was selected to minimize fatigue and cybersickness, it limits the ability to assess how users adapt to and learn to use SA cues over time. It is possible that longer-term use would lead to habituation, changes in perceived distraction, or more efficient information processing strategies. Longitudinal studies are needed to better understand these dynamics.

Fourth, the tasks used in this study—passive viewing in Study 1 and simplified interactive tasks in Study 2—were designed to control cognitive demands and isolate the effects of SA cue design. However, these tasks do not fully capture the complexity of real-world VR applications, such as collaborative work, multitasking, or high-stakes decision-making environments. As a result, the extent to which these findings translate to more complex and dynamic VR contexts remains an open question.

Fifth, we relied on established psychometric instruments (e.g., SART, SMS) to ensure comparability with existing human factors literature. While these are standardized benchmarks, they were not originally developed for immersive contexts. High levels of VR immersion may influence how users interpret specific scale items; therefore, future research should prioritize the development and validation of psychometric tools specifically tailored for XR technology.

14.1 Future work

Moving forward, the next steps could involve conducting studies to investigate the effect of customization on users’ experiences in VR environments. By allowing participants to customize SA cues based on their preferences, task requirements, and environmental constraints, researchers can explore how tailored interventions impact users’ perception, comprehension, and projection of information. Additionally, future studies could consider including participants from different age groups to examine potential age-related differences in information needs and VR experience. The studies in this paper mainly involved participants of college age, and their characteristics may differ from other demographic groups, potentially influencing their responses to SA cues in VR settings. By including participants from diverse age groups, researchers can gain a more comprehensive understanding of how factors such as age, living arrangements, and technological literacy shape users’ experiences and needs in VR environments.

Going back to the positive computing framework, wellbeing in VR consists of multiple dimensions (Calvo and Peters, 2014). This paper mainly focused on multitasking and mindful attention to the VR task as they relate to the mindfulness dimension. Another avenue for future work is to delve deeper into other aspects of wellbeing in VR beyond mindfulness. Positive emotions play a crucial role in shaping users’ experiences and perceptions of VR applications. Moreover, investigating motivation and engagement in VR environments is crucial for maximizing users’ enjoyment and productivity. Furthermore, exploring the implications of self-awareness in VR can lead to the development of applications that promote mental health and wellbeing. It is important to address these aspects to ensure that VR experiences are not only immersive but also enriching and supportive of users’ mental and emotional health.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/journals/human-dynamics/articles/10.3389/fhumd.2026.1654475/full#supplementary-material

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Footnotes

Edited by: Genovefa Kefalidou, University of Leicester, United Kingdom

Reviewed by: Max Allen Teaford, University of Tennessee at Chattanooga, United States

Phillip Tretten, Luleå University of Technology, Sweden