Effects of augmented reality landmark visualization on incidental spatial knowledge acquisition in autonomous driving: a virtual reality experiment

1 Introduction

Autonomous driving is on a path to transforming the transportation industry and our travel experiences. Global investment in autonomous driving already exceeds 80 billion US dollars and is expected to increase at an annual growth rate of 35% over the next 10 years (Sun and Huang, 2023; Precedence Research, 2023). According to McKinsey and Co., by 2035 autonomous driving will generate 300–400 billion US dollars in revenue, and global sales of autonomous vehicles will reach 15–20 million units (Deichmann et al., 2023; Chai et al., 2021). As an emerging technology for future transportation, autonomous driving aims to improve traffic conditions, reduce energy consumption, and alleviate commute stress, ultimately replacing human drivers in operating vehicles across various road conditions (Yurtsever et al., 2020). The SAE International Standard (SAE International, 2021) defines six levels of driving automation, ranging from no automation (Level 0) to full automation (Level 5). At the highest SAE levels, vehicles are capable of driving themselves without any human intervention under certain (e.g., highway; Level 4) or all conditions (Level 5). However, nowadays even the most advanced autonomous driving systems, such as Tesla Autopilot and Mercedes-Benz Drive Pilot, are classified as Level 2 or 3 (partial or conditional automation). At these levels, the car can control most aspects of driving, but the human driver must remain aware of road conditions and be ready to take over at any time (Level 2) or when alarmed by the system (Level 3; Clemens, 2023).

1.1 Cognitive mapping under increased automation

Although autonomous vehicles are not expected to replace all driving tasks in the coming years, increased automation can relieve human drivers of tedious maneuvering operations and make driving safer and easier. This development, however, raises concerns about human adaptation to new technologies, particularly regarding the psychological and behavioral challenges drivers may face. One such concern is rooted in previous research on traditional (non-autonomous) driving, which has investigated the impact of GPS-based navigation aids on cognitive maps.

Cognitive maps are mental representations of the relative locations and attributes of phenomena in spatial environments (Downs and Stea, 1973). They contain spatial knowledge necessary for understanding our living environment and therefore play a pivotal role in everyday functioning, such as wayfinding and decision-making. Furthermore, cognitive mapping skills have been shown to correlate with success in many professional occupations, including those of geologists, surveying engineers, and architects, where precise spatial orientation and wayfinding in large-scale environments are crucial (Ishikawa, 2023). Despite their importance, drivers who rely on turn-by-turn navigation instructions do not need to construct accurate cognitive maps of their surroundings, as they only need to follow directions provided by the navigation system. Numerous studies have revealed the detrimental effects of GPS use on spatial knowledge acquisition (Dahmani and Bohbot, 2020; Hejtmánek et al., 2018; Parush et al., 2007). For example, individuals who frequently rely on GPS guidance tend to have a smaller posterior hippocampus (the brain region associated with spatial memory) than those with extensive navigation experience, such as London Taxi drivers (Maguire et al., 2000). Habitual GPS use has also been linked to a steeper decline in spatial learning ability over time (based on a 3-year longitudinal follow-up; Dahmani and Bohbot, 2020).

Compared to GPS-based navigation, driving an autonomous car requires even less engagement of human cognition and action. Consequently, the negative impacts observed in previous studies of traditional driving may become more pronounced in the era of autonomous driving. Once a car reaches automation Level 3, drivers are no longer required to steer, brake, or accelerate in most situations. Under such conditions, spatial knowledge acquisition becomes largely incidental, developing not through deliberate effort but as an unintended byproduct of passive observation when moving through an environment. However, reduced interaction with environmental input can disengage autonomous drivers from actively perceiving and understanding their surroundings. Over the long term, this disengagement may impair the construction of cognitive maps and lead to negative consequences in situations where automation fails or human intervention becomes necessary (Qin and Karimi, 2020; Brishtel et al., 2021; Brügger et al., 2019). Given the importance of cognitive mapping for daily functioning and the challenges posed by both GPS-assisted navigation and future autonomous driving, a key question is how cognitive mapping skills, particularly spatial knowledge acquisition and retrieval, can be supported and improved in scenarios involving increased levels of automation.

1.2 Landmarks

To mitigate the degradation of spatial knowledge, new generations of navigation aids are being designed to enrich spatial cues that are critical for cognitive mapping. As a basic form of spatial cues, landmarks are distinctive features that can be easily recognized and remembered in everyday settings (Stankiewicz and Kalia, 2007). Two functional types of landmarks, local and distant, have been widely studied for their role in improving the accuracy of mental representations.

Local landmarks, such as bus stations and grocery stores, are visible only within a limited range and from specific viewpoints. They are often incorporated into route directions (e.g., at turning points), serving as intermediate goals for individuals navigating through unfamiliar neighborhoods (Steck and Mallot, 2000). In contrast, distant landmarks, such as skyscrapers and mountain peaks, are visible from far away and tend to remain stable as an individual moves short distances. These landmarks therefore function as global reference points that anchor the spatial relationships among locations stored in a cognitive map (Steck and Mallot, 2000; Liu et al., 2022).

Empirical studies have demonstrated the benefit of enriching one’s surroundings with landmark-based information. For instance, associating local landmarks along a travel route with photorealistic 3D icons on mobile maps or with personalized audio instructions during navigation (e.g., “here you can buy your favourite book”) has been shown to facilitate spatial learning (Gramann et al., 2017), especially for individuals with lower spatial abilities (Kapaj et al., 2024). Similarly, visualizing distant landmarks located off-route at the edge of a mobile screen contributes to spatial orientation and a configurational understanding of large-scale environments (Li, 2020; Li and Zhao, 2017). Overall, these findings highlight the effectiveness of landmark cues in counteracting the negative consequences of reliance on navigation aids, particularly when exploring large and unfamiliar areas.

1.3 AR-HUD

Inspired by the beneficial role of landmarks in spatial learning, in our previous study Li (2023) developed an augmented reality head-up display (AR-HUD) that visualizes landmark information on the windshield of autonomous vehicles. The concept of a head-up display (HUD) is not new; it was first employed in aviation to allow pilots to access essential flight parameters while maintaining a forward view. Later, HUDs were introduced as part of in-vehicle display systems to enhance driving performance and safety. For example, many recent BMW car models are equipped with a HUD that presents dashboard information, such as vehicle speed and lane deviation, at the bottom portion of the windshield (Maroto et al., 2018). With recent advances in real-time spatial tracking and computer vision, augmented reality (AR) has matured as a visualization technology capable of accurately integrating virtual objects into the real world. Consequently, the combination of AR and HUD technologies (AR-HUD) enables digital elements to be projected directly into the driver’s visual field and seamlessly aligned with the surrounding physical environment (Zhang, 2021).

AR-HUDs have the potential to facilitate the development of cognitive maps. Due to their installation position, conventional in-vehicle displays require drivers to shift their attention away from the forward road view. Such divided attention has been identified as a major limitation of using smartphones or center-stack displays for navigation and spatial learning (Kim and Dey, 2009). In autonomous vehicles, although drivers are still required to monitor the environment, their motivation to acquire spatial knowledge through indirect visual interfaces may be further reduced. In this regard, AR-HUDs enable a seamless integration of digital and physical spaces, thereby minimizing the need to switch attention between the primary driving scene and secondary display views (Kim and Dey, 2009). Moreover, AR-HUDs provide a unified interface in which digital spatial information can be presented in a world-fixed reference frame that is dynamically aligned with road conditions. Viewing virtual and physical objects within a shared reference frame is expected to support more accurate encoding and recall of spatial relationships compared to separated or screen-based presentations.

Despite a recent upswing in research applying AR-HUDs to autonomous driving, existing studies have predominantly focused on vehicle monitoring (Tijerina et al., 2016), situational awareness (Feierle et al., 2019), risk perception (Kim et al., 2023), and entertainment (Riegler et al., 2019), while largely neglecting the spatial cognition of autonomous drivers. To the best of our knowledge, our previous study (Li, 2023) was the only one to empirically evaluate the effectiveness of AR-HUDs in supporting cognitive mapping during autonomous driving. In that study, a novel display system was proposed that visualized both local and distant landmarks on the windshield. Results from an online experiment with 60 participants indicated that landmark cues overlaid on the windshield (AR landmarks) enhanced awareness of on-route features as well as relative directions to distant locations.

However, several limitations should be noted in the earlier study. First, all AR landmarks were pre-rendered, and the autonomous driving experience was simulated using prerecorded real-world driving videos. These videos were presented on a two-dimensional (2D) computer screen with a narrow and fixed field of view, which prevented participants from changing their viewing direction or interacting with the simulated environment. Second, the use of real-world scenes constrained the experimenter’s ability to manipulate or control environmental elements, thereby reducing the flexibility and reusability of the stimuli. Third, the online experimental setup reduced procedural control and may have allowed undesired participant behavior such as inattention or cheating (Palan and Schitter, 2018). Finally, all participants experienced the normal windshield condition before the AR-HUD, meaning that learning effects cannot be ruled out. This fixed order is particularly problematic for assessing incidental spatial learning, as prior exposure to the environment may have influenced performance in the AR-HUD condition independently of the interface manipulation.

1.4 The present study

To address the limitations outlined above, the present study employs virtual reality (VR) in a laboratory setting to simulate the use of an AR-HUD in autonomous vehicles. VR is a technology in which users wear a head-mounted display (HMD) to experience and interact with a computer-generated environment while being isolated from their physical surroundings. By leveraging real-time graphics rendering, stereoscopic vision, and motion tracking, VR enables users to immerse themselves in a realistic three-dimensional (3D) space and explore virtual objects in ways that closely resemble real-world behavior (Slater et al., 2010). Over the past decade, VR has increasingly been applied to support cognitive processes through evidence-based design (e.g., Kalantari and Neo, 2020); however, the validity of user data generated in VR remains a topic of ongoing debate (Sanchez et al., 2022). In this context, construct validity refers to the extent to which an experimental intervention accurately captures the theoretical or empirical constructs it is intended to measure (Harris et al., 2020). If VR simulations can faithfully represent autonomous driving and AR-HUD experiences, the current experimental setup should be able to reproduce differences between conditions consistent with findings from our previous work (Li, 2023), as well as results from related research domains.

In addition, recent technological advances have enabled the integration of eye tracking into off-the-shelf HMDs, eliminating the need for external webcams or specialized gaze-tracking devices. Compared to our earlier study, which relied solely on task performance measures (Li, 2023), bringing eye tracking into a VR environment allows for the collection of substantially richer data on participants’ visual attention and perceptual strategies when interacting with highly controlled yet ecologically valid stimuli, as well as on the underlying cognitive processes involved (Adhanom et al., 2023). The combined use of VR and eye tracking in a laboratory setting is therefore expected to provide deeper insights into the psychological and behavioral impacts of AR-HUDs, and in particular into why certain visualization designs are effective while others are not.

The overarching goal of the present research is to use the experimental data obtained from a VR-based simulation to validate the effectiveness of the AR-HUD previously developed by Li (2023) in enriching landmark information during autonomous driving. Specifically, this study aims to:

• Examine the effect of visualized AR landmarks on the windshield on the incidental acquisition of different levels of spatial knowledge;
• Examine whether road types, such as highways and local roads, moderate the effect of AR landmarks on spatial knowledge acquisition;
• Explore how the gaze behavior of autonomous drivers differs across the experimental conditions described above.

The first two objectives build directly on our previous study (Li, 2023) and serve both to replicate earlier findings and to assess the construct validity of the VR-based simulation. As in Li’s study (2023), we included local road and highway environments as two ecologically distinct autonomous driving scenarios that differ in the relative availability of local and distant environmental cues. The local road environment contains richer local landmarks along the route, which indicate turning points and provide salient cues for route following (Steck and Mallot, 2000), whereas the highway environment provides more opportunities to use off-route distant landmarks as stable reference cues for spatial updating and maintaining global orientation (Steck and Mallot, 2000; Liu et al., 2022). Therefore, the manipulation of road type was intended to explore whether the effectiveness of our AR-HUD depends on the dominant type of landmark cue in the environment. The third objective leverages the unique capabilities of eye tracking to elucidate participants’ perceptual and attentional processes, thereby helping to explain observed differences in spatial cognitive performance across conditions.

2 Materials and methods

2.1 Design and participants

The study employed a two-by-two between-subjects design with two independent variables: windshield display (with vs. without AR landmarks) and road type (local road vs. highway). A total of 68 adult participants were randomly assigned to one of four experimental conditions: local road with AR landmarks (17 participants; 12 females; average age 25.5 years, SD = 3.30), local road without AR landmarks (17 participants; 10 females; average age 25.2 years, SD = 2.70), highway with AR landmarks (17 participants; 13 females; average age 25.5 years, SD = 2.53), and highway without AR landmarks (17 participants; 14 females; average age 26.2 years, SD = 4.71). All participants were recruited from the Department of Environmental Sciences at Wageningen University and Research. Participants were eligible to take part if they had normal or corrected-to-normal vision, had sufficient English proficiency to understand the experimental instructions, were willing to wear a VR HMD, and reported no severe vestibular disorders. Participants with corrected-to-normal vision were asked to wear contact lenses rather than glasses during the experiment to ensure proper HMD fitting and improve the quality of the eye-tracking data. Of these participants, 50 were full-time students, 13 were university employees, and five were not currently employed.

2.2 Materials and apparatus

The virtual local road and the virtual highway environments used in this study were modeled and rendered in Unity3D. Building models were obtained from the Fantastic City Generator Unity asset, and textures were extracted from the video frames used in our previous study (Li, 2023; Figures 1A,B). Both virtual environments were replicated from real urban locations in the United States to enhance authenticity and provide a naturalistic experience for participants. The virtual local road environment consisted of six on-route (local) landmarks (basketball court, gas station, museum, Albany Street, park, and church) and four off-route (distant) landmarks (mountain, harbor, downtown, and Guilerland). The virtual highway environment consisted of two on-route (local) landmarks (gas station and rest area) and six off-route (distant) landmarks (Cohoes, Troy, Menands, Watervliet, toll booth, and Albany; Figures 1C,D). These local landmarks were indicated by building signage placed directly in front of them, while distant landmarks were indicated by traffic signs displaying their names and directions placed along the roads in the virtual environment.

An autonomous driving simulation was implemented using the Realistic Car Controller from the Unity Asset Store. For each environment, a set of waypoints was defined at road turning points, producing a curved trajectory for the vehicle to follow at a predefined speed. This resulted in an autonomous driving duration of approximately 3 minutes for the virtual local road (speed range: 25–55 km/h) and 4 minutes for the virtual highway (speed range: 40–90 km/h). During the simulated autonomous driving, no pedestrians or other vehicles were presented on the roadway to avoid potential confounds. A first-person VR camera was positioned at the driver’s seat, allowing participants to view both the car interior and the surrounding environment through the windshield and windows (Figures 1E,F).

The AR-HUD system was developed in Unity3D based on our previous design (Li, 2023). Figure 1E illustrates how different regions of the virtual windshield were assigned to display distinct types of information. In this system, landmarks located in the car’s heading direction were visualized as representative symbols (graphics and/or text) overlaid on the windshield. The visualization position of each landmark on the windshield was dynamically determined as a function of the landmark’s distance from the driver. A threshold of 1 km was used as a pragmatic visibility-based criterion to distinguish landmarks that could be directly perceived within the driver’s immediate surroundings from those that were too far away to be directly visible. Landmarks located beyond the visible region (>1 km from the driver’s location) were classified by the AR-HUD as distant landmarks and visualized in the upper portion of the windshield. As the car approached a landmark, once the distance fell below 1 km, the landmark was reclassified as a local landmark and projected onto the central portion of the windshield. Furthermore, when a local landmark was passed by the car, it was transformed into an off-screen landmark and remained visible along the left or right edge of the windshield until it moved completely outside the car’s heading range. For distant and off-screen landmarks, windshield symbol transparency increased as the distance between the driver and the corresponding environmental feature increased. To avoid cognitive overload and potential visual clutter, a maximum of five landmarks, up to two local and three distant landmarks, could be displayed simultaneously on the windshield. When more landmarks were present in the environment, only those closest to the driver were visualized. This design choice is consistent with the findings of Cheng et al. (2023); Cheng et al. (2022), who reported that displaying five landmarks on a mobile map, compared to three or seven landmarks, improved spatial learning along a route without exceeding users’ cognitive capacity during navigation in urban environments. In addition, auxiliary information, including the current street name and speed limit, was displayed in the bottom portion of the windshield.

The simulation was conducted using the Varjo Aero HMD to provide an immersive VR experience while recording eye-tracking data. The HMD features binocular eye tracking at 200 Hz with sub-degree accuracy, a horizontal field of view of 115°, and a display resolution of 2880 × 2720 pixels per eye with a maximum refresh rate of 90 Hz. The HMD was connected to a desktop PC running Windows 10, equipped with an Intel i9-11900 KF Processor, 32 GB of RAM, and an NVIDIA RTX 3050 graphics card.

2.3 Procedure and measures

Participants were recruited voluntarily through online and offline advertisements around the campus of Wageningen University and Research. Upon arrival at a dedicated VR laboratory and after providing informed consent, participants first completed a demographic questionnaire (e.g., age and gender), followed by the Santa Babara Sense of Direction Scale (SBSOD). The SBSOD consists of 15 items rated on a five-point Likert scale (adapted from the original seven-point scale) and assesses individuals’ self-reported general spatial ability (Hegarty, 2002); higher mean SBSOD scores indicate a better perceived sense of direction. Overall, participants reported a moderate sense of direction (M = 2.97, SD = 0.68).

Participants were then seated in a non-swivel chair, fitted with the Varjo Aero HMD, and asked to complete one of four VR simulated autonomous driving scenarios, defined by a 2 × 2 between-subjects design with road type (local road vs. highway) and windshield display (with vs. without AR landmarks). Prior to each scenario, participants were instructed to experience an autonomous driving simulation in VR and to pay attention to the surrounding environment in case of automation failure. However, they were not explicitly instructed to memorize landmark locations or their spatial relationships. In the conditions without AR landmarks, participants simply sat in a moving car with a normal windshield while experiencing the virtual environment. In conditions with AR landmarks, participants saw landmark icons and text projected onto the windshield via the AR-HUD and were provided with an explanation of the visualization design using a graphical illustration (similar to Figure 1E) prior to the VR experience. During the experience, participants could freely move their heads to explore the virtual environment; no hand controllers were provided due to the passive nature of the task. For a detailed comparison of experimental conditions, readers are referred to the video demonstrations provided in the Supplementary Material.

2.3.1 Eye-tracking measures

Eye-tracking data were recorded continuously throughout the VR experience. To examine participants’ visual attention to the windshield display, windshield-based areas of interest (AOIs) were defined. Specifically, the central portion and the left and right edge regions of the windshield were combined into a single AOI, and raw eye-movement data were exported separately for the top, middle, and bottom windshield areas. Fixations were identified using the Identification by Velocity Threshold (I-VT) algorithm with a minimum fixation duration of 150 m and a velocity threshold of 130°/s (i.e., a fixation was identified when eye movement velocity remained below 130°/s for at least 150 m; Olsen, 2012; Duchowski et al., 2002). From the fixation data, two windshield-based eye-tracking metrics commonly used in visual cognitive research were calculated (Dong et al., 2022; Kapaj et al., 2021; Enders et al., 2021; Holmqvist et al., 2011):

• Average fixation duration: The mean duration of all fixations within a windshield AOI (in milliseconds). Longer average fixation durations may indicate greater difficulty in interpreting visual information (Hooge and Erkelens, 1998; Just and Carpenter, 1976).
• Fixation frequency: The number of fixations per minute within a windshield AOI. Higher fixation frequencies have been associated with more efficient visual search behavior (Dong et al., 2022).

Because the AR-HUD was designed to influence spatial knowledge acquisition through interaction with environmental features, four additional landmark-based eye-tracking metrics were calculated based on fixations on individual landmarks within the environment. Local landmarks referred to distinctive environmental features located along the traveled route (see Figures 1C,D), and fixations were calculated directly on these objects. In contrast, most distant landmarks, except for the mountain and downtown in the local road environment, were not visually accessible due to distance; therefore, fixations on traffic signs indicating the names and directions of these landmarks were used as proxies for gaze behavior related to distant landmarks.

• Time-to-first-fixation rank: For each landmark, participants were ranked according to how quickly they fixated on that landmark after the start of the autonomous driving scenario, from fastest (rank = 1) to slowest. Rank was used instead of raw time-to-first-fixation values because the raw values were computed from the start of the application and were therefore strongly influenced by when each landmark was encountered along the route. Lower ranks indicate earlier fixation on a landmark, which may reflect stronger bottom-up attentional capture driven by its visual salience (Rothkopf et al., 2007; Skripkauskaite et al., 2023).
• First fixation duration: The duration of the initial fixation on a landmark (in milliseconds). Longer first fixation durations may reflect deeper initial perceptual or cognitive processing of the landmark (Holmqvist et al., 2011, 521–523; Peake et al., 2020).
• Log total fixation duration: The cumulative duration of all fixations on a landmark, log-transformed using the natural logarithm to reduce the influence of extreme values arising from substantial individual differences in repeated viewing behavior, thereby improving comparability across participants. Higher values may indicate more sustained attentional engagement and increased cognitive processing demands associated with the landmark (Hvelplund, 2014).
• Proportion of landmarks fixated: The proportion of landmarks that received at least one fixation relative to the total number of landmarks in an environment. Higher proportions reflect more extensive visual exploration of the environment.

2.3.2 Spatial knowledge acquisition

After completing the autonomous driving scenario, participants’ spatial knowledge acquisition was assessed using three multiple-choice questions adapted from Li (2023). The first question measured route-level knowledge by asking participants to recall the correct sequence of three landmarks along the traveled route, with three local landmarks on the local road or two local and one distant landmark on the highway. Responses were scored as incorrect, partially correct (two landmarks in the correct order), or correct. The second question assessed directional knowledge by asking participants to associate the names of two distant landmarks with their corresponding directions relative to the traveled route on a simplified map; responses were similarly categorized as incorrect, partially correct (one correct match), or correct. Depending on the experimental condition, landmarks were indicated either via AR-HUD overlays or through building and traffic signage in the environment. The third question assessed configurational knowledge by asking participants to select the correct route configuration from three alternatives. Each alternative presented a simplified map of the same study area, with the start and end points and a possible route trajectory highlighted. Participants were asked to select the route that best matched the one they had experienced during autonomous driving, and their responses were scored as incorrect (0) or correct (1). These questions were designed based on the Landmark-Route-Survey Knowledge framework, which describes how individuals acquire, organize, and use spatial information during navigation (Siegel and White, 1975). In addition to accuracy, the time spent answering each spatial knowledge question was recorded for analysis. The full set of questions and response options is provided in the Supplementary Material.

2.3.3 Simulator sickness

After the experiment, participants rated simulator sickness using two five-point Likert scale items (Helland et al., 2016): (1) “How much general discomfort are you experiencing right now?” and (2) “To what extent did you experience simulator sickness during the virtual experience?” Responses were averaged to yield an overall sickness score (1 = not at all; 5 = very uncomfortable). Overall, participants reported only slight simulator sickness (M = 1.55, SD = 0.77).

2.4 Data analysis

All analyses were conducted in RStudio (version 2023.12.1). For accuracy in each spatial knowledge task (route, directional, and configurational), we performed a two-way analysis of deviance (ANODE). Ordinal regression models were fitted for route and directional task accuracy using the clm function from the ordinal R package, whereas a binary regression model was fitted for configurational task accuracy using the glm function from the stats package. Windshield type (AR-HUD vs. normal windshield), road type (local road vs. highway), and their interaction were included as fixed effects. Participants’ age, gender, and SBSOD score were included as covariates. Post hoc comparisons were conducted only for significant interactions. The proportional odds assumption for ordinal regression models was tested using nominal effects tests. This assumption was violated, indicating that the effects of some predictors may vary across different levels of the corresponding spatial knowledge measure. However, partial proportional odds models resulted in singularities and non-identifiable parameters; therefore, proportional odds models were retained to ensure stable estimation. These models estimate a single odds ratio (OR) for each predictor. Consequently, the reported ORs should be interpreted as overall associations rather than as evidence that the effects are consistent across all outcome levels. For the binary logistic regression model, assumptions including independence of observations and absence of separation were evaluated and met.

For the completion time of each spatial knowledge task (route, directional, configurational), a two-way analysis of covariance (ANCOVA) was conducted using the lm function from the stats package, with windshield type and road type as between-subject factors and participants’ age, gender, and SBSOD score as covariates. Normality and homogeneity of variance assumptions were assessed using skewness and kurtosis indices. When these assumptions were violated, robust linear regression analyses were conducted using the limrob function from the robustbase package.

For each windshield-based eye-tracking measure (average fixation duration and fixation frequency) in each windshield area (top, middle, and bottom), a two-way ANCOVA was conducted with windshield type and road type as between-subject factors and age, gender, and SBSOD score as covariates. Robust linear regression was applied when assumptions were violated.

Finally, for each landmark-based eye-tracking measure (time-to-first-fixation rank, first fixation duration, total fixation duration, and proportion of landmarks fixated), a separate one-way ANCOVA was conducted for each combination of landmark type (local versus distant) and road type (local road versus highway). Windshield type was included as the between-subject factor, with age, gender, and SBSOD score entered as covariates. Robust linear regression was used when assumptions were violated.

3 Results

3.1 Spatial knowledge task accuracy

3.1.1 Route task accuracy

There was a significant main effect of windshield type, χ ²(1) = 6.78, p = 0.009, OR = 7.31, 95% confidence interval (CI) [1.74, 36.3], indicating that participants using the AR-HUD achieved significantly higher accuracy in the route knowledge task than those using the normal windshield (Figure 2, top left). The main effect of road type, χ ²(1) = 0.05, p = 0.82, OR = 1.18, 95% CI [0.29, 4.75], and the interaction between windshield type and road type, χ ²(1) = 0.18, p = 0.67, OR = 1.66, 95% CI [0.17, 19.8], were not significant. Among the covariates included in the model (age, gender, and SBSOD score), only age showed a significant effect, χ ²(1) = 5.38, p = 0.02, OR = 0.83, 95% CI [0.70, 0.97], indicating that higher age was associated with lower route task accuracy.

3.1.2 Directional task accuracy

The main effect of windshield type was significant, χ ²(1) = 5.37, p = 0.02, OR = 4.72, 95% CI [1.31, 18.4], indicating that participants using the AR-HUD achieved significantly higher accuracy in the directional knowledge task than those using the normal windshield (Figure 2, top right). The main effect of road type, χ ²(1) = 1.05, p = 0.31, OR = 2.02, 95% CI [0.53, 7.96], and the interaction between windshield type and road type, χ ²(1) = 1.08, p = 0.30, OR = 0.38, 95% CI [0.06, 2.36], were not significant. None of the covariates (age, gender, and SBSOD score) showed significant effects (ps > 0.17).

3.1.3 Configurational task accuracy

There was a significant interaction between windshield type and road type, χ ²(1) = 4.21, p = 0.04, OR = 11.7, 95% CI [1.11, 161.3] (Figure 2, bottom). Holm corrected simple-effects pairwise comparisons showed that AR-HUD (vs. normal windshield) significantly improved configurational task accuracy on highways (OR = 11.7, z = 2.50, p = 0.01), but not on local roads (OR = 1.004, z = 0.005, p = 1.00). In addition, participants using the AR-HUD showed significantly higher configurational task accuracy on highways than on local roads (OR = 22.0, z = 3.20, p = 0.001). This difference was not observed for participants using the normal windshield (OR = 1.89, z = 0.81, p = 0.42). None of the covariates showed significant effects (ps > 0.11).

3.2 Spatial knowledge task completion time

3.2.1 Route task completion time

The main effect of windshield type was significant, F(1, 61) = 13.2, p < 0.001, η p 2 = 0.19, indicating that participants using the AR-HUD spent significantly less time completing the route knowledge task than those using the normal windshield (Figure 3, top left). The main effect of road type, F(1, 61) = 0.44, p = 0.51, η p 2 = 0.02, and the interaction between windshield type and road type, F(1, 61) = 0.14, p = 0.71, η p 2 < 0.01, were not significant. None of the covariates (age, gender, and SBSOD score) showed significant effects (ps > 0.30).

3.2.2 Directional task completion time

Results from the robust linear regression showed that neither windshield type, b = −6.88, standard error (SE) = 6.59, t = −1.04, p = 0.30, η p 2 < 0.01, nor road type, b = −6.44, SE = 6.80, t = −0.95, p = 0.35, η p 2 = 0.05, significantly predicted directional task completion time. Their interaction was also not significant, b = −1.38, SE = 9.17, t = −0.15, p = 0.88, η p 2 < 0.01 (Figure 3, top right). Among the covariates included in the model, only participants’ age was a significant predictor, b = 1.63, SE = 0.62, t = 2.61, p = 0.01, indicating that older participants took longer time to complete the directional knowledge task.

3.2.3 Configurational task completion time

The main effect of windshield type was significant, F(1, 61) = 6.37, p = 0.01, η p 2 = 0.08, indicating that participants using the AR-HUD spent significantly less time completing the configurational knowledge task than those using the normal windshield (Figure 3, bottom). The main effect of road type was also significant, F(1, 61) = 4.61, p = 0.04, η p 2 = 0.11, indicating a shorter completion time on local roads than highways. The interaction between windshield type and road type was not significant, F(1, 61) = 0.01, p = 0.92, η p 2 < 0.01. Among the covariates, gender was the only significant predictor, F(1, 61) = 7.64, p = 0.008, with female participants taking longer time to complete the configurational knowledge task than male participants.

3.3 Windshield-based eye-tracking behavior

3.3.1 Top windshield area

For average fixation duration, there was a significant main effect of road type, F(1, 61) = 4.31, p = 0.04, η p 2 = 0.06, indicating longer fixation durations in the highway environment than in the local road environment within the top windshield area (Figure 4A, left). The main effect of windshield type, F(1, 61) = 0.38, p = 0.54, η p 2 < 0.01, and the interaction between windshield type and road type, F(1, 61) = 0.83, p = 0.37, η p 2 = 0.01, were not significant. None of the covariates (age, gender, and SBSOD score) showed significant effects, ps > 0.45.

With respect to fixation frequency, there was a significant interaction between windshield type and road type, F(1, 61) = 10.9, p = 0.002, η p 2 = 0.15 (Figure 4A, right). Holm-corrected simple-effects comparisons showed that the AR-HUD, compared with the normal windshield, significantly increased fixation frequency on highways, t(61) = 3.41, p = 0.001, Cohen’s d = 1.22, but not on local roads, t(61) = −1.20, p = 0.23, Cohen’s d = −0.42. Furthermore, participants using the normal windshield exhibited significantly lower fixation frequency on highways than on local roads, t(61) = −5.01, p < 0.001, Cohen’s d = −1.80, whereas this difference was not observed when using the AR-HUD, t(61) = −0.48, p = 0.63, Cohen’s d = −0.17. None of the covariates showed significant effects, ps > 0.30.

3.3.2 Middle windshield area

For average fixation duration, there was a significant main effect of road type, F(1, 61) = 25.8, p < 0.001, η p 2 = 0.29, indicating that fixation duration was significantly longer in the highway environment than in the local road environment (Figure 4B, left). The main effect of windshield type, F(1, 61) = 0.40, p = 0.53, η p 2 < 0.01, and the interaction between windshield type and road type, F(1, 61) = 0.01, p = 0.91, η p 2 < 0.01, were not significant. None of the covariates showed significant effects, ps > 0.42.

With respect to fixation frequency, robust linear regression analyses revealed a significant main effect of road type, b = −11.5, SE = 3.63, t = −3.17, p = 0.002, η p 2 = 0.06, indicating higher fixation frequency on local roads than on highways (Figure 4B, right). The main effect of windshield type, b = −5.05, SE = 4.61, t = −1.09, p = 0.28, η p 2 = 0.01, and the interaction, b = −0.99, SE = 6.55, t = −0.15, p = 0.88, η p 2 = 0.03, were not significant. None of the covariates showed significant effects, ps > 0.44.

3.3.3 Bottom windshield area

For average fixation duration, robust linear regression analyses revealed a significant interaction between windshield type and road type, b = −289.7, SE = 122.3, t = −2.37, p = 0.02, η p 2 = 0.06 (Figure 4C, left). Holm-corrected simple-effects comparisons indicated that the highway environment was associated with significantly longer average fixation durations than the local road environment for both the normal windshield, t(61) = 4.20, p < 0.001, Cohen’s d = 1.78, and the AR-HUD, t(61) = 2.86, p = 0.006, Cohen’s d = 0.83. In addition, the AR-HUD, compared with the normal windshield, significantly reduced average fixation duration on highways, t(61) = −2.19, p = 0.03, Cohen’s d = −0.90, but not on local roads, t(61) = 0.15, p = 0.88, Cohen’s d = 0.05. Among the covariates included in the model, only age was a significant predictor, b = −20.7, SE = 9.19, t = −2.25, p = 0.03, indicating that older participants exhibited shorter average fixation durations in the bottom windshield area.

With regard to fixation frequency, there was a main effect of windshield type, F(1, 61) = 14.4, p < 0.001, η p 2 = 0.20, indicating that participants using the AR-HUD showed significantly higher fixation frequency in the bottom windshield area than those using the normal windshield (Figure 4C, right). The main effect of road type was also significant, F(1, 61) = 4.77, p = 0.03, η p 2 = 0.07, indicating higher fixation frequency on highways than on local roads. The interaction between windshield type and road type was not significant, F(1, 61) = 0.02, p = 0.89, η p 2 < 0.01. None of the covariates showed significant effects, ps > 0.53.

3.4 Landmark-based eye-tracking behavior

For local landmarks (i.e., distinctive environmental features located along the road), participants using the AR-HUD fixated on these landmarks earlier, as indicated by significantly lower time-to-first-fixation ranks, and exhibited significantly longer first fixation durations compared to participants using the normal windshield, in both local road and highway environments. No significant differences were observed between the AR-HUD and normal windshield for log total fixation duration or for the proportion of landmarks fixated (Table 1).

TABLE 1. Summary of time-to-first-fixation rank, first fixation duration (ms), log total fixation duration, and proportion of landmarks fixated on local landmarks, comparing participants using the AR-HUD and the normal windshield in local road and highway environments.

Estimated marginal means (EMMeans) and standard errors (SEs) were adjusted for relevant covariates (age, gender, and SBSOD, score). For the proportion of landmarks fixated in the highway environment, the planned ANCOVA, could not be performed because the normal windshield condition exhibited a ceiling effect (all values = 1). Raw descriptive means and SEs, are therefore reported, and a one-sample Wilcoxon signed-rank test comparing the AR-HUD, condition against the reference value of one indicated no significant difference (V = 0, p = 1.00). Significance levels are indicated as p < .05 (*), p < .001 (***).

For distant landmarks (i.e., road signs indicating the names and directions of distant landmarks), there were no significant differences between the AR-HUD and normal windshield conditions for time-to-first-fixation rank, first fixation duration, log total fixation duration, or the proportion of landmarks fixated, in either local road or highway environments (Table 2).

TABLE 2. Summary of time-to-first-fixation rank, first fixation duration (ms), log total fixation duration, and proportion of landmarks fixated for distant landmarks, comparing participants using the AR-HUD and the normal windshield in local road and highway environments.

EMMeans, and SEs, were adjusted for relevant covariates (age, gender, and SBSOD, score).

4 Discussion

In the route knowledge task, participants using the AR-HUD demonstrated better performance, reflected in higher accuracy and shorter completion time, compared with those using a normal windshield. Route knowledge refers to the sequential ordering of landmarks encountered along a path (Siegel and White, 1975). Distinct environmental features along a route, such as individual buildings, are readily perceived as local landmarks; however, these features are often limited in size and visible only from a short distance (Steck and Mallot, 2000). During autonomous driving, as the car rapidly approaches and passes through its surroundings, multiple visual cues along and around the roadway dynamically and simultaneously enter the driver’s visual field. Given the limited capacity of working memory for processing and maintaining information (see cognitive load theory; Sweller, 2011), drivers may struggle to attend to or correctly identify local landmarks. To mitigate potential cognitive overload, our AR-HUD highlights nearby environmental features using graphical symbols overlaid on the central portion of the windshield. By increasing the visual salience of route-relevant landmarks, this design may help drivers more effectively acquire and encode spatial cues from their environment.

Directional knowledge concerns the relative directions of locations within an environment. The results of this study showed that participants using the AR-HUD during autonomous driving demonstrated significantly improved directional knowledge across a large spatial extent (beyond the path directly traveled by car) regardless of road type. This improvement appears to stem from the AR-HUD’s ability to provide continuous directional cues through the persistent visualization of distant landmarks in the top windshield area. These findings align with previous research demonstrating that highlighting distant landmarks within the field of view (Liu et al., 2022) or placing off-screen landmarks at the edges of mobile screens (Li, 2020; Li and Zhao, 2017) supports individuals’ ability to maintain directional awareness during movement. Distant landmarks, such as mountains and skyscrapers, are stable and visible over long distances and can therefore serve as reliable directional anchors, allowing drivers to repeatedly calibrate their heading and relative orientation while progressing through the environment (Steck and Mallot, 2000). Importantly, in the present study, this benefit extends even to landmarks that were not directly visible at a given moment. Their consistent representation in terms of identity and direction via the AR-HUD likely supports more accurate directional judgments by offering a stable reference frame. Although such landmarks are often indicated by traffic and guide signs along the road, drivers may not readily recognize them or encode their spatial significance. By making directional information explicit and continuously available, the AR-HUD may therefore facilitate the construction of more reliable spatial relationships among distant environmental features, rather than requiring drivers to infer their relative locations indirectly from intermittent signage.

Configurational knowledge involves understanding the layout of route segments and the metric interrelationships between them (Curtis et al., 1981). Chrastil and Warren (2012) indicate that active exploration of an environment plays a crucial role in configurational learning. During autonomous driving, drivers passively experience movement along a prescribed path with predetermined turns. Such passive visual exposure eliminates most of the physical and cognitive activities that are deemed necessary for active exploration, such as motor control of action and navigational decision making (see Chrastil and Warren, 2012 for a comprehensive overview), and thus may hinder the development of configurational knowledge.

Consistent with this expectation, participants in our study generally exhibited low configurational knowledge accuracy, particularly when using the normal windshield (mean accuracy <0.40). With respect to the use of the AR-HUD, our results suggest that its impact on the acquisition of configurational knowledge is modulated by road type. In the local road environment, configurational knowledge accuracy remained low regardless of windshield type. Unlike route and directional knowledge tasks, the configurational knowledge task does not rely on landmark representations but instead assesses metric relations, including distances and directions, between road segments. As shown in Figures 1C,D, the local road environment contains substantially more local landmarks than the highway environment (six vs. two). Given that participants were instructed to pay attention to their surrounding environment, the rich positional cues along the local road may have discouraged the encoding of metric information sufficient for path integration, thereby leading to reduced configurational knowledge accuracy. Several previous studies lend support to this interpretation. For example, Hornsey and Hibbard (2024) discussed the dual role of landmarks (or “shadows” in their study), either as anchors or distractors when estimating metric information such as distance. Moreover, a VR study conducted by Foo et al. (2007) found that, when reliable information about environmental locations is present, people tend to adopt the computationally simpler landmark-based strategy, relying on local landmarks to navigate and infer novel shortcuts rather than developing path-integrated metric mental representations of the environment.

In contrast, in the highway environment, which is characterized by fewer local landmarks and more distant landmarks, participants demonstrated significantly higher configurational knowledge accuracy when using the AR-HUD compared to the normal windshield. In line with the general idea from previous studies on environmental features that are visible from far away (Lin and Chang, 2025; Credé et al., 2020; Liu et al., 2022), a possible explanation is that visualizing the relative directions of distant landmarks on the AR-HUD helps autonomous car drivers maintain a global orientation. This, in turn, provides a stable cardinal reference frame within which different route segments can be integrated into a coherent mental representation of the spatial layout of the environment.

Although the distinct functions of local versus distant landmarks in spatial learning provide theoretically plausible explanations, the local road and highway virtual environments differed not only in landmark composition but also in other characteristics inherent to the two road types, including travel speed, duration, number of turns, and geometric layout. Importantly, this study treated road type as a holistic, ecologically meaningful environmental variable and was therefore not designed to isolate the contribution of each of these characteristics. Consequently, interpretations that rely on landmark composition alone to account for the interaction observed in configurational knowledge performance should be considered tentative until stronger empirical evidence controlling for these environmental characteristics becomes available.

Regarding the task completion time, participants using the AR-HUD spent significantly less time on the configurational knowledge task than those using the normal windshield. This finding suggests that the AR-HUD may facilitate more efficient processing of environmental information relevant to the development of configurational knowledge. As a result, participants aided by the AR-HUD may have experienced greater confidence during the task.

Overall, the results discussed above partially align with the findings of our previous study (Li, 2023). Both studies indicate a positive effect of landmark visualization on the AR-HUD for the acquisition of route and directional knowledge. However, the pattern of results differs for configurational knowledge. While the previous study showed that configurational knowledge was primarily influenced by road type (with higher accuracy on local roads than highways), the current study reveals a positive effect of the AR-HUD on configurational knowledge specifically in the highway environment. A key difference between the two studies lies in the experimental medium. The previous study employed real-world driving videos and was conducted via an online crowdsourcing platform, whereas the current study used VR technology in a laboratory to simulate the autonomous driving experience. While our current approach affords realistic interactions under a highly controlled experimental procedure (Zhao et al., 2021), the similar effects observed for route and directional knowledge across both studies suggest satisfactory validity and robustness of these findings across different media and experimental settings. In contrast, the divergent effects observed for configurational knowledge may reflect an interaction between experimental medium and road type. Compared to watching desktop-based videos online, the immersive visual experience provided by VR HMDs offers richer and more precise optical flow information, which may facilitate distance estimation and, in turn, support better configurational understanding in relatively sparse environments such as highways (Zhao et al., 2020; Campos et al., 2007). Future research is needed to further disentangle these effects.

According to the eye-tracking results across windshield areas (top, middle, and bottom), participants exhibited longer average fixation durations when traveling on highways than on local roads across all areas of the windshield. Similarly, participants using a normal windshield showed lower fixation frequencies (and therefore lower information search efficiency) in the top and middle windshield areas on highways than on local roads. Together, these patterns may reflect greater difficulty in searching for and processing spatial information while traveling on highways, potentially due to the relative scarcity of local landmarks (Holmqvist et al., 2011, 483–577; Just and Carpenter, 1976; Dong et al., 2022). In contrast, participants using the AR-HUD showed a gaze pattern consistent with quicker and more efficient information search in the top windshield area, as reflected by their higher fixation frequency (Dong et al., 2022), compared with those using a normal windshield. This finding, together with the superior performance of AR-HUD participants in the configurational knowledge task in the highway environment and in the directional knowledge task (which assessed the relative directions of distant landmarks), suggests that our AR-HUD design, particularly the visualization of distant landmarks, has the potential to support the incidental acquisition of spatial knowledge in environments with limited on-route or local environmental features (e.g., highways and open oceans; Darken and Cevik, 1999).

Moreover, participants assisted by the AR-HUD exhibited longer average fixation durations and higher fixation frequency in the bottom windshield area (used to display auxiliary information such as speed limits) when traveling on highways than on local roads. This pattern may reflect the fact that sparse environments such as highways provide limited visual cues for distance perception (Hornsey and Hibbard, 2024). As a result, participants in these environments may experience greater uncertainty regarding their travel velocity and therefore allocate more attention to monitoring speed-related information. Conversely, this increased engagement with metric cues (e.g., speed limit) may help explain why participants using the AR-HUD developed more accurate configurational knowledge on highways compared with the local road environment.

The eye-tracking results for local landmarks suggest that the AR-HUD may facilitate the incidental acquisition of spatial knowledge by efficiently guiding attention toward task-relevant environmental features. Compared with the normal windshield, participants using the AR-HUD fixated on local landmarks earlier, as reflected by lower time-to-first-fixation ranks, and exhibited longer first fixation durations, which may indicate enhanced bottom-up attentional capture followed by deeper initial cognitive processing of these landmarks (Peake et al., 2020; Skripkauskaite et al., 2023). Importantly, these benefits were not accompanied by increases in log total fixation duration, which suggests that the AR-HUD altered the timing and quality of local landmark processing rather than the overall amount of visual inspection of the environment (Hvelplund, 2014). Hilton et al. (2025) demonstrated that increasing the visual similarity between landmark icons shown on a mobile map and their real-world referents facilitates automatic perceptual matching, as reflected in enhanced parietal P200 amplitude in electroencephalography, thereby reducing the mental translation cost of integrating information between the map display and the environment. Our AR-HUD visualization of local landmarks addresses this matching process more directly by spatially co-registering graphical cues with their corresponding environmental features, so that the symbol and its referent are perceived simultaneously within the same visual field. A similar visuospatial alignment mechanism was also implemented by Kim and Dey (2009), who projected prospective road configurations onto the windshield to support older drivers’ navigation. The eye-tracking results of our study suggest that such alignment may enable the rapid identification and effective interpretation of key spatial information without increasing sustained cognitive effort. This provides a plausible explanation for the superior route knowledge performance (assessing the sequence of visits to on-route or local landmarks) observed in AR-HUD participants and appears well suited to supporting incidental spatial learning in the context of autonomous driving.

In contrast, no significant differences in landmark-based gaze behavior were observed between AR-HUD and normal windshield conditions for distant landmark cues. This suggests that the benefits of distant landmark visualization, as reflected in the superior performance of AR-HUD participants on the directional knowledge task, may not arise from increased visual interaction with environmental proxies such as road signs. However, it is important to note that, in the highway environment, which was dominated by distant landmark cues, participants using the AR-HUD showed a significantly higher fixation frequency in the top windshield area than those using the normal windshield. This suggests that the distant landmark visualization in the AR-HUD successfully attracted visual attention and may have supported more efficient information search at the interface level (Dong et al., 2022). Taken together, the eye-tracking results at both the interface and environment levels suggest that, compared with the direct spatial co-registration of local landmark visualization, the directional and identity information of distant landmarks presented on the top windshield area of the AR-HUD may support incidental spatial learning through a different mechanism, namely by providing a coarse but stable, globally anchored representation of spatial relations among distant landmarks (Liu et al., 2022; Li, 2020). Such information may improve the understanding of relative directions and, potentially, the acquisition of configurational knowledge.

In addition to the main effects of AR-HUD and road type, several individual difference effects associated with the covariates also emerged. Older participants showed lower route knowledge accuracy and required more time to complete the directional knowledge task. Although this finding is broadly consistent with prior research suggesting age-related decline in certain aspects of spatial learning and memory (Head and Isom, 2010), the relatively restricted age range of our sample limits the generalizability of this interpretation. Female participants required more time to complete the configurational knowledge task, although no gender difference was observed in task accuracy. In contrast, the self-report sense of direction (measured by the SBSOD) did not significantly predict performance in any analysis. Because this study was not designed to investigate individual differences, these findings should be interpreted with caution. Nevertheless, future research may benefit from examining how AR-HUD design interacts with more diverse user characteristics, such as cultural background, previous driving experience, and more specific measures of spatial ability, to jointly influence incidental spatial learning performance. Such investigations would require substantially larger sample sizes to provide sufficient statistical power for unravelling potential interaction effects.

5 Limitations and future directions

This study has several limitations. First, the AR-HUD system used in this study was limited in two major respects. On the one hand, the system displayed a maximum of five landmarks simultaneously (up to two local and three distant landmarks). Although this design was intended to reduce visual clutter and cognitive overload, it may also have constrained the amount of spatial information available to participants. To uncover its full potential, future research should systematically examine how the number and density of AR landmark cues, along with individual differences such as age and working memory capacity, jointly shape spatial learning outcomes. On the other hand, the AR-HUD focused exclusively on the projection and localization of landmarks, whereas other forms of spatial information, such as road structure and distances between locations, may also contribute to cognitive mapping (e.g., Kim and Dey, 2009). Future research could incorporate these additional visio-spatial cues into AR-HUD designs and explore different presentation strategies, such as hierarchical organization of information, to maintain cognitive load at a manageable level (Billinghurst et al., 2015).

Second, spatial learning was assessed using a single multiple-choice question for each type of spatial knowledge. This approach was adopted to remain consistent with our previous work (Li, 2023) and to enable direct comparisons across studies. With only one item, responses may be disproportionately influenced by participants’ memory of the specific spatial information being tested. The small number of response categories also provides limited variation in the knowledge scores. Although single-item measures reduce the reliability and sensitivity of performance assessment, the validity of our conclusions is strengthened by the inclusion of eye-tracking metrics that captured participants’ visual attention and processing of landmark information during VR exposure. Nonetheless, future studies should complement brief knowledge checks with more sophisticated spatial knowledge measures, such as sketch map drawing or landmark pointing tasks (e.g., Zhao et al., 2023), to reduce measurement noise and reveal subtler differences in drivers’ cognitive maps across conditions.

Third, the virtual local road and highway environments used in this study were modeled after real-world settings in the United States, and landmarks were selected based on surveys with local residents. Although one environment primarily featured local landmarks and the other featured distant landmarks, both types of landmarks were present in each virtual environment. This overlap limits our ability to draw definitive conclusions about the relative contributions of local versus distant landmarks to incidental spatial learning. In addition, because the landmarks were derived from real-world settings, their visual properties (e.g., shape, size, and salience) were not systematically controlled, which may have introduced confounding factors influencing gaze behavior and spatial learning performance (Raubal and Winter 2002). Future studies would benefit from using more controlled virtual environments that explicitly manipulate the presence and AR visualization of local and distant landmarks while keeping landmark characteristics constant, to enable more robust and precise causal inferences.

Fourth, our VR experience simplified several aspects of the real-world driving environment. In particular, pedestrians, surrounding traffic, and sound were excluded to minimize experimental confounds. In reality, these competing perceptual and cognitive demands may influence how much attention drivers allocate to AR landmarks and environmental features, potentially affecting the observed benefits of the AR-HUD for incidental spatial learning. Future research should therefore validate these findings in more complex simulations and, ultimately, real-world autonomous driving scenarios.

Fifth, eye-tracking measures were calculated based on fixations and interpreted in a relatively direct manner, for example as indicators of search efficiency or the depth and difficulty of cognitive processing. However, such measures are not process-pure and can be influenced by multiple factors, including oculomotor variability (e.g., ocular fatigue; Friedman and Komogortsev, 2024), individual cognitive differences (e.g., working memory capacity; Loh et al., 2023), and task demands (e.g., aesthetic judgment versus memorization; Loh et al., 2023). Although we controlled for individual characteristics (age, gender, and sense of direction) and accounted for task performance when interpreting the eye-tracking results, future work could refine these interpretations by incorporating complementary measures such as mental workload (e.g., NASA Task Load Index; Hart and Staveland, 1988), confidence or usability ratings, scanpath-based visual exploration strategies, and time-on-task effects to better triangulate the underlying cognitive processes.

Sixth, fixation detection in this study relied on the I-VT algorithm with a velocity threshold of 130°/s. This threshold was adopted based on prior work using HMDs for 3D eye movement analysis (Duchowski et al., 2002). However, VR eye-tracking research is still in its early stages, and most fixation identification algorithms and parameter settings were originally developed and validated in traditional desktop-based paradigms (e.g., reading or viewing static 2D images; Hessels et al., 2018). In immersive virtual environments, gaze behavior is shaped by additional factors such as head-eye coordination, body movement, and depth perception, which may systematically influence the performance of fixation identification algorithms derived from desktop setups. Future research is therefore needed to reevaluate and calibrate I-VT parameters to ensure their suitability for measuring visual attention in VR applications (see preliminary work by Zhao et al., 2025).

Finally, the relatively small sample size (N = 68, n = 17 per experimental condition), combined with the inclusion of multiple predictors and covariates in the regression models, may have reduced the stability and precision of the estimated effects, particularly for the interactions. In addition, the sample consisted predominantly of young university students and included a relatively high proportion of females, which may limit how our results can be applied to the broader population of autonomous car users, particularly middle-aged and older adults. Future studies should therefore recruit larger and more demographically diverse samples to assess the robustness and generalizability of the present findings.

6 Conclusion

As autonomous driving technology becomes increasingly prevalent, drivers may face growing challenges in developing spatial cognition and forming cognitive maps while monitoring, rather than actively steering, the vehicle through the environment (Qin and Karimi, 2020; Brishtel et al., 2021). To explore how these challenges might be addressed, we conducted an autonomous driving experiment in immersive VR, investigating how AR-HUD and environmental characteristics jointly influence incidental spatial learning and gaze behavior. The results show that landmark information presented on the AR-HUD facilitates the acquisition of route and directional knowledge. In contrast, its contribution to configurational knowledge is modulated by road type, with the AR-HUD outperforming a normal windshield only in sparse environments, such as highways with few local landmarks. With respect to gaze behavior, our findings suggest that highways impose greater cognitive demands than local roads due to the lack of distinctive environmental cues. The AR-HUD may help drivers cope with these demands by enabling more efficient information search, particularly through the visualization of distant landmarks in the top windshield area, which supports the continuous representation of global orientation. At the same time, local landmark information projected onto the middle windshield area of the AR-HUD may guide attention to corresponding environmental features, facilitating their earlier and deeper initial processing.

The present work makes two major contributions. First, to our knowledge this is the first study to use immersive VR simulation to assess the benefits that AR-HUDs may provide to autonomous car drivers in terms of spatial knowledge acquisition for cognitive map development. By comparing the present findings with those from our earlier study of a similar design using online videos recorded in real-world environments (Li, 2023), we show that both studies consistently point to the beneficial role of our AR-HUD design, which incorporates the visualization of both local and distant landmarks, in supporting the incidental spatial learning of autonomous car drivers, particularly with respect to route and directional-level knowledge. This convergence across experimental media provides evidence suggesting the construct validity of VR as a research tool for investigating spatial cognition and the potential consequences of human adaptation to emerging technologies. Although the differences observed in certain aspects of configurational knowledge acquisition require further investigation, the overall results support the use of VR simulation for studying in-vehicle information systems in scenarios that are currently difficult or even impossible to realize in real-world or traditional video-based settings (e.g., implementing gesture- or gaze-based interactions, visualizing 3D traffic information in its native form, or systematically manipulating route or environmental features). Second, this study integrates eye-tracking analysis with immersive VR to examine driver visual attention and its relationship to the cognitive processing of different environmental features and AR-HUD design elements. The combination enables a more fine-grained understanding of how spatial information is perceived, attended to, and encoded at the individual level, thereby offering insights into the mechanisms through which the AR-HUD supports incidental spatial learning. As such, the integration of eye tracking and VR holds strong potential to inform the evidence-based design of future spatial user interfaces for autonomous driving.

Generative AI statement

The author(s) declared that generative AI was used in the creation of this manuscript. During the preparation of this work, the author(s) used ChatGPT to check grammar and paraphrase for enhanced English and make mistake-free and clear writing.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Gen Li, Nanjing Forestry University, China

Reviewed by: Ziyad Aldoski, Duhok Polytechnic University, Iraq

Zhicheng Dai, Beijing Jiaotong University, China