Adaptive just-in-time training supports skill retention in virtual reality simulations

Effective training systems must be designed to support not only the mastery of trained skills, but also retention, especially in high consequence industries. As trainee time is finite, it is critical to optimize parameters of frequency, spacing, and content facilitation to enable efficiently retaining skills. These constraints have driven the growth of just-in-time (JIT) training approaches, with targeted reinforcement of skills just prior to evaluation. In this study, we implement and assess a virtual reality (VR) adaptive difficulty facilitation algorithm for providing personalized skill retention training. We also investigate the cadence of refresher training. N = 31 subjects trained to proficiency in a VR spacecraft piloting simulation task. They were then assigned to one of three retention conditions: no training for a 5-week period, once-per-week training with an adaptive difficulty algorithm, or a single JIT training session prior to evaluation. Final evaluation performance was affected by training condition (p = 0.02), with better retention performance with JIT training than with no training (p = 0.018). In contrast, the subjects in the weekly trained condition may suffer from overtraining at high difficulty levels, with a negative correlation in a post hoc analysis between difficulty level achieved and final evaluation performance (p = 0.02). Our assessments of flow, motivation, preparation, and confidence were not significantly impacted by retention training condition. The results of this study demonstrate the value of personalized, adaptive JIT in VR for retaining training and mitigating skill loss in a virtual reality simulation environment.

In many of today’s modern working environments, the ability to successfully perform tasks is predicated on the ability to retain skills across a gap between when an individual has been trained at a task and when they must execute it. A positive correlation between skill decay and time elapsed since training has been well established (Childs et al., 1983; Childs and Spears, 1986; Hollister et al., 1973), though certain tasks appear to be more susceptible to decay than others (Arthur et al., 1998). In aerospace contexts, skill decay is anticipated to be exacerbated due to increased automation in the cockpit (Volz and Dorneich, 2020). In a 1972 study of manual control training in simulators, procedural skills decayed to “unacceptable” levels only 1 month after the completion of training (Sitterley and Berge, 1972). These trends continue into the modern era, such as skill decay observed in pilots who spend less time in manual command of aircraft due to automation (Casner et al., 2014) and in those who experienced a reduction in flight hours during the COVID-19 pandemic (Curran and Pulver, 2024). Skill retention becomes a vital requirement to shape the design of training paradigms to support durable retention of skill (Healy et al., 2014).

Refresher training is an essential countermeasure to attenuate skill decay (Sitterley and Berge, 1972; Klostermann et al., 2022). Symbolic rehearsal, a method of refresher training involving visualization of task performance without physical execution, has been proposed as a low-fidelity and easy-to-implement option for skill maintenance (Farr and Farr, 1987). However, practice with the complete task has been shown to outperform symbolic rehearsal (Kluge et al., 2012), demonstrating the need for simulations and recreations of physical tasks for the purpose of refresher training. An additional study on refresher training found practicing the task supports skill retention more so than either symbolic rehearsal or skill testing (Frank and Kluge, 2019). Observations of task-specific differences in susceptibility to decay (Vlasblom et al., 2020) suggest that the ideal paradigm of refresher training to support retention may be unique to the skill in question.

Just-in-time (JIT) training is a popular concept proposed for refreshing skills with minimal burden on trainee time. This method of training is enticing for industries where learners must gain and maintain large volumes of skill, such as in spaceflight applications where it has been used for medical skills training in astronauts on the International Space Station (ISS) (Fo et al., 2005). However, JIT trainers must be carefully designed to refresh high-impact skills in a short amount of time, which requires identifying the most critical skills at risk of decay and scoping the training accordingly (Barshi and Dempsey, 2016), as well as timing the presentation of information in a way that reflects how the task is executed (Kester et al., 2001). As a result, precise delivery of impactful JIT training often relies on human facilitators to perform fine-tune adjustments in the moment to increase learning impacts, a method frequently used when JIT training is conducted in healthcare settings (Miller et al., 2025).

Adaptive systems pose a unique opportunity to replace operator facilitation of JIT trainer construction, as an efficient algorithm can assess a user’s current skill level and rapidly respond to new acquisition of skill during training (Zohaib, 2018; Lopes and Lopes, 2022). Especially during dynamic learning and re-learning of skill, adaptive systems may foster the challenge-skill balance necessary to support a cognitive state of flow (Engeser and Rheinberg, 2008). Higher levels of flow may improve performance during simulation-based training, an effect that is strengthened by self-efficacy (Pavlas et al., 2010). JIT adaptive interventions, newly described as JITAI (Nahum-Shani et al., 2018), are an emerging practice that has been primarily explored in the context of behavioral health management and support. Features of JITAI include real time, personalized support when a user is at risk of engaging in unwanted behavior, triggered by data passively collected from a user in order to provide a dynamic response. Notably, this response is not controlled or initiated by the user, but algorithmically by the system itself (Hardeman et al., 2019). However, Hardeman et al.‘s review found a lack of compelling evidence of JITAI effectiveness, suggesting that the emerging field has many gaps to fill in the design and testing of evidence based JITAI platforms. While JITAI adapts when to intervene, there has been less exploration of using a similar method to adapt what content to deliver. Additionally, adaptive JIT interventions have yet to be explored in settings of professional skill development or high consequence industries, such as for aerospace or medical operators.

The ideal cadence and design of refresher training remains to be seen, especially for maintaining complex skills that carry a risk of significant operational consequences in the face of inadequate human performance. Human spaceflight missions are an example where significant refresher training needs exist, as long duration exploration missions (LDEM) such as a mission to Mars require astronaut crews to retain skill across significant gaps of time between training and performance. Many of the tasks that crew are expected to perform are mission critical, meaning that even the smallest mistake or forgotten step due to skill decay can lead to severe mission consequences. The spaceflight environment of LDEM also requires innovative training solutions to comply with constraints in onboard mass, volume, and power as well as round-trip communication delays that prevent facilitated training support. Virtual reality (VR) systems provide a solution that meets those design constraints, providing an immersive platform that can be used for multi-scenario training. As a result, we chose to explore the impact of an adaptive, just-in-time paradigm of refresher training in virtual reality (VR) within the context of maintaining mission-critical piloting skills in a spaceflight entry, descent, and landing (EDL) task. Our primary research question was to understand how refresher training cadence impacts the loss of EDL skills when there are delays between training and execution, and understand subsequent effects on the operator’s confidence and training experience.

Subjects

This research was approved by the Institutional Review Board (IRB) at the University of Colorado Boulder (Protocol #23–0097). All participants provided written informed consent to participate in the study. A total of n = 31 subjects were recruited for testing and upon successfully completing proficiency training (described below), were randomly assigned to one of 3 refresher training conditions: “Weekly” adaptive VR training, just-in-time (“JIT”) adaptive VR training, or “No Training”. A summary of subject demographics can be found in Table 1 in the results. Subjects between the ages of 18–65 with vision correctable to 20/20 and no form of colorblindness were included in the investigation, and subjects who scored above the 90th percentile on a motion sickness susceptibility questionnaire (MSSQ) (Golding, 1998) were candidates for exclusion, but no subjects were excluded.

Entry descent and landing task

The task used in this investigation was a spaceflight Entry, Descent, and Landing (EDL) task onto the surface of Mars (see Verniani et al., 2024 for more information on task scoring and adaptive difficulty design). Briefly, this task is comprised of three subtasks that a participant must complete sequentially within a given trial. First, the landing site (LS) subtask asks subjects to survey a topographic map to select a landing site that optimizes distance to sites of scientific significance while avoiding steep terrain. Next, they must fly the vehicle to the point of interest in the manual control (MC) subtask. Subjects input pitch and roll commands to follow a guidance reticle on their primary flight display and null wind disturbances. In the final terminal descent (TD) subtask, participants use a hand thruster to control their vertical descent speed as they approach the surface, balancing the need for a soft touchdown with a limited fuel supply for controlling this speed.

Across the sessions in this experiment, the EDL task was presented in a combination of difficulty modalities and immersive formats. First, in the “Adaptive” difficulty modality, task difficulty changed dynamically based on user performance using a 2-up, 1-down (2U1D) unlocked training algorithm where each subtask difficulty changed independently (Verniani et al., 2024). Second, in the “Static” difficulty modality, the difficulty remained constant at level 12 out of a possible 25. During proficiency and refresher training, participants completed the task in VR with an HTC Vive Pro head mounted display. During timepoint 1 (T1) and timepoint 2 (T2) evaluation sessions, participants performed the task in a fixed-based physical cockpit simulator at the University of Colorado at Boulder. The physical cockpit simulator can be seen in Figure 1. The joystick controls, seating configuration, task dynamics, etc., were matched between VR and the physical cockpit simulator, except for the means of visual presentation and immersiveness of performing the task in a physical cockpit.

Experimental protocol

All participants were first trained in VR to a baseline level of proficiency on the EDL task to normalize starting skills before investigating the impact of refresher training. Participants were shown a video that introduced the goals and procedure of the task. After this video, they were given up to 40 trials to attempt to reach proficiency, defined as performing 2 consecutive excellent trials where all three subtasks were at a difficulty level of 12. Task difficulty started at level 8 out of a possible 25 in all 3 subtasks and adapted based on user performance using the 2U1D algorithm described in section B. 35 subjects were recruited, but not included from final analysis either because of a failure to reach proficiency (in which case they did not proceed to Sessions 2–8, n = 31) or in a few cases (n = 4) a failure to return for all study sessions, and in one individual because due to suffering from cybersickness.

Baseline performance was then evaluated to verify proficiency and measure skill in a fashion similar to what a spaceflight crewmember might be assessed to have at a “preflight” timepoint (T1) before training conditions diverged. This evaluation took place between 18–48 h after the proficiency session. Subjects were tested in the physical simulator described in section B, where they completed 5 trials of static difficulty at level 12 out of 25.

The next phase of the study sought to replicate a spaceflight mission scenario, where a crewmember may experience a significant gap in time between initial training and task performance during which skill retention was threatened. The objective of this study was to examine differences in skill maintenance and training experience between three methods of refresher training throughout this period.

Subjects in the Weekly training condition came for a session of 10 training trials, once per week for 5 weeks. They trained in VR, where the 2U1D unlocked algorithm adjusted difficulty based on performance. Weekly sessions were bound to an 18–48-h window of the exact 1-week timepoint of the preceding visit. In the JIT training group, subjects only returned for one refresher training session, 5 weeks after their preflight baseline and 18–48 h before their postflight evaluation. This session also consisted of 10 trials using the adaptive difficulty algorithm in VR. Subjects in the No Train condition did not return for any additional training between preflight and postflight evaluation, serving as a control group representing skill loss for this task when no skill refresher interventions are implemented. This experiment design is demonstrated visually in Figure 2.

Approximately 5 weeks after their proficiency training (mean = 37.3 days, SD = 2.2 days), all subjects returned for a second timepoint (T2) evaluation session. Like the T1 evaluation baseline, subjects were tested in the physical simulator described in section B. They completed 5 trials of static difficulty at level 12 out of 25 regardless of performance in their 5-week inflight training period.

Prior to each evaluation and refresher session, subjects completed a survey of sleep duration and reported their perceived motivation and preparation by answering the following questions on a 7-point Likert scale. “ I feel motivated to give my best performance possible ”

“I understand the task procedures and feel prepared to begin.”

They also reported their confidence of task success on a scale from 0 to 100, where reports from 0 to 49 indicated a belief that a participant was more likely to fail than succeed, reports of 50 indicating equal probability, and reports from 51 to 100 indicated a belief that a participant was more likely to succeed than fail. At the end of each data collection session across all phases of the experiment, subjects completed a survey report of their flow state using the Flow Short Scale survey (Engeser and Rheinberg, 2008).

Measures and statistical methods

Data analysis was divided according to two major analyses to address our research question of interest. The first question explored objective performance outcomes¸ where we investigated whether an adaptive JIT trainer adequately mitigates the loss of EDL skills when there are delays between training and performance. We hypothesized that JIT training would sufficiently mitigate decays of skill seen in subjects who had no refresher training between simulated preflight and postflight data collections and would provide equivalent final performance when compared to the weekly refresher training group.

The second analysis explored subjective training experience, where we explored whether an adaptive JIT modality could foster confidence and flow by creating appropriate challenge-skill balance while encouraging user compliance by reducing time burdens associated with weekly training. We hypothesized that participants in weekly training conditions would report the highest perceived confidence of performance success during postflight evaluation, followed by JIT participants who had a lack of opportunity for over-practice, and the lowest confidence of success in participants who had no training during the simulated inflight phase. For the construct of flow, we hypothesized that adaptive difficulty would overcome repetition fatigue to support calibrated flow in both weekly trained participants and participants in the JIT condition. We expect participants in the no refresher training condition to report lower experiences of flow due to high perceived challenge and low perceived skill during postflight evaluation.

Significance was assessed at the alpha = 0.05 level and all assumptions for each statistical test were evaluated to determine whether parametric or non-parametric hypothesis testing was appropriate.

Performance: For all analyses, performance on the EDL task was scored using an integrated skill metric (ISM) (Verniani et al., 2024). In this metric, subtask performance is first compared to best possible outcomes to create a ratio score from 0 to 1 using a linear transformation with an error function anchored to the definitions of poor, adequate, and excellent performance provided as feedback to the user during training and ensuring crashes are scored as a 0 (Verniani et al., 2024). Subtask scores are then summed to create a total performance score that ranges from 0 to 3. That sum is divided by the highest possible performance outcome (3) and multiplied by 100 to create an interpretable metric representing integrated skill as a ratio of maximum possible performance ranging from 0 to 100 as shown in Equation 1.

First, we verified that preflight training successfully normalized subjects to equal levels of task proficiency before training conditions diverged through examination of two metrics. No significant difference was observed between groups in the number of trials needed to reach proficiency (ANOVA F (2,28) = 1.94, p = 0.16). Evaluation of T1 skill confirmed equivalent baseline performance between groups (ANOVA F (2,28) = 0.09, p – = 0.91).

After confirming the consistency of the refresher training, we analyzed performance in the evaluation phase. While 5 trials were collected during each evaluation session, the primary outcome of the objective performance research question explored performance on only the first T2 trial. In a true spaceflight mission where EDL skill must be performed successfully, crewmembers will have only one opportunity to execute a successful landing. To investigate this real-world outcome, we isolated and analyzed the first trial T2 performance as an indicator of task success separately from the T2 five trial average, which may be subject to re-learning effects. This metric failed to meet assumptions of normality, resulting in the use of non-parametric Kruskal–Wallis hypothesis testing. Post-hoc analysis was performed using Dunn’s multiple comparison test.

As a secondary outcome measure, the percent change in skill between pre and postflight was analyzed to investigate the magnitude of skill retention across training conditions. T1 skill was characterized for each subject by taking a 5-trial average of the ISM as performed during that evaluation session. The 5-trial average was selected to provide a baseline of subject skill that was less sensitive to noise. T2 performance was represented by 1 st trial skill to capture the real-world consequence described above. A retention metric for each subject was then calculated according to Equation 2.

Data met assumptions of normality and equal variance, leading to hypothesis testing via one-way ANOVA, with post hoc comparison using Tukey’s HSD.

Training experience: The primary outcome of the subjective experience analysis was flow during the T2 evaluation session, reported via survey taken at the conclusion of the session. Flow survey results are analyzed as total flow experience scored by averaging all item responses, as well as broken into survey subdimensions of perceived task importance, skill, demands, and skill-demand balance, per the survey’s published scoring method (Engeser and Rheinberg, 2008). Secondary outcomes for this research question included motivation, preparation, and confidence during the T2 evaluation session, reported via survey taken prior to beginning the first trial in the session. In both cases, data failed to meet assumptions for parametric testing, thus nonparametric testing was performed through Kruskal–Wallis rank sum testing.

Investigating potential negative training: While not an original hypothesis of this work, we explored a negative training effective that was further investigated through correlation analysis. Spearman rank correlation was conducted to assess the relationship between difficulty achieved during inflight training and performance on the 1 st postflight trial for the weekly training conditions.

Demographic data as seen in Table 1 shows no significant difference in demographic distributions between groups for any parameter including sex, age, previous flight experience, or previous VR experience. Over 50% of subjects had no previous experience with either simulator or aircraft piloting, while only 3 subjects (9.7%) had any experience in command of a physical aircraft.

Performance

Evaluating our primary hypothesis, a significant difference in first trial T2 performance was seen between training groups (Kruskal χ 2 (2) = 7.86, p = 0.02). Dunn’s post hoc with a Holm adjustment for multiple comparisons revealed a significantly higher performance for JIT subjects (median = 97.4, interquartile range (IQR) = 19.5) as compared to No Train (median = 65.0, IQR = 19.2) subjects (adjusted p = 0.018), but no significant difference between Weekly VR (median = 79.1, IQR = 30.4) compared to either JIT or No Train groups. This result can be seen in Figure 3.

A significant difference in skill retention metric was observed between groups (ANOVA F (2,28) = 3.97, p = 0.03, η 2 = 0.22). Tukey post hoc comparison revealed significantly higher retention of skill in the JIT group (median = 11.1, IQR = 24.5) as compared to No Train (median = −23.9, IQR = 19.1) subjects (adjusted p = 0.02). When the skill metric is calculated where T2 skill is captured by a five-trial average instead of trial 1 performance, this effect is lost (ANOVA F (2,28) = 1.85, p = 0.175).

Two-tailed t-tests with a Holm adjustment for multiple comparisons were performed on each group to test skill retention against a null hypothesis of 0, which would indicate skill did not change between pre- and postflight evaluation. No Train subjects showed significant loss of skill (median = −23.9, IQR = 19.1, t (9) = −3.79, adjusted p = 0.013). Skill change for JIT (median = 11.1, IQR = 24.5) and Weekly (median = −11.2, IQR = 46.1) subjects was not significantly different than zero (adjusted p = 0.63 and 0.55, respectively). This result can be seen in Figure 4.

Subjective experience

Table 2 shows flow survey responses during T2 evaluation. There was no significant difference in overall flow experience between training conditions (Kruskal–Wallis p = 0.3), nor in any flow submetrics (Kruskal–Wallis p > 0.1 for all submetrics). However, subjects in the JIT condition reported the highest overall flow, perceived skill, and perceived importance of the task. Subjects in the no train condition reported the lowest perceived skill and the highest perceived difficulty.

Table 3 shows subjective reports of motivation, preparation, and confidence during T2 evaluation. Similarly to flow, no significant differences were seen across training conditions in reports of motivation, preparation, or confidence as seen in Table 3. Notably, the weekly condition had the lowest confidence of success for T2 evaluation, reporting lower than participants who engaged in no refresher training.

Negative training

Finally, in a post hoc analysis, we evaluated whether participants who trained in VR at higher difficulty level had better or worse performance on T2 trial 1 after their training. No significant relationship between training difficulty level and T2 trial 1 performance was seen for JIT training group (Spearman ρ = −0.15, p = 0.66), however a negative correlation was found for the Weekly VR group (Spearman ρ = −0.71, p = 0.02) as seen in Figure 5.

Our findings demonstrate that adaptive JIT training can successfully mitigate skill loss seen during a 5-week gap between training and task execution in a complex, operationally relevant task. Interestingly, we did not find any effect of refresher training on subjective flow and confidence, although data trended toward supporting JIT training over the other two refresher training approaches. In a post hoc analysis, we found a statistical relationship linking the highest level of training difficulty for those who trained weekly with their performance, suggesting that overtraining can also be a problem when too much or too difficult refresher training than necessary is provided. These results are novel in that they are some of the first application of adaptive JIT to these kinds of training environments. Further, they support prior literature on the importance of refresher training to mitigate skill decay.

An important aspect of this study that leads to its ecological validity in the training of this kind of task is the manner by which we implemented the training paradigm. To isolate the impact of training cadence on the retention of skill, the reduction of between-subject variance in baseline skill was essential. Proficiency training was intended to accomplish this normalization before exposure to various refresher training interventions (see Figure 2). The concept of proficiency training also mimics the reality of astronaut training, where crew must demonstrate sufficient skill through extensive training as an astronaut candidate before receiving a flight assignment. It was desired that proficiency training occur within one session, therefore proficiency and evaluation difficulty was set at level 12 as pilot testing revealed that subjects could not consistently reach proficiency at difficulty levels higher than 12 within a single 40-trial session. No significant difference in T1 baseline integrated skill was seen between groups (p = 0.91). As such, subjects across all training groups were found to be proficient, and equivalent in skill, before entering the “retention” phase of the experiment where training intervention was varied. Equilibration of baseline skill is a critical requirement to facilitate investigation of true changes in retention between training conditions (Ackerman and Tatel, 2024). Failure to establish this baseline has been cited as a limitation in previous investigations of medical procedural skill retention (Ansquer et al., 2019; Higgins et al., 2020).

Performance

In analysis of postflight performance, No Train subjects were found to have a significant loss of skill over 5 weeks without access to training (p = 0.01). This result is expected and matches previous literature on the loss of learned skill over time without maintenance training (Childs et al., 1983; Childs and Spears, 1986; Hollister et al., 1973; Arthur et al., 1998). The impacts of negative training on the performance of subjects in the Weekly VR group may have contributed to no significant difference in skill maintenance between Weekly VR and No Train groups. However, the Weekly VR group was not significantly different than a skill retention of 0 (i.e., no loss or gain of skill during the 5 week period) (p = 0.55). In contrast, the JIT training group showed a group average improvement in skill between pre and postflight evaluation, demonstrating significant mitigation of skill decay as compared to no train subjects (p = 0.02).

Negative training experienced by subjects in the Weekly VR condition may have contributed to the lower observed skill during postflight performance. A few subjects in the Weekly VR condition showed more decline in skill than some subjects who experienced no training in the 5-week period, suggesting that negative training may have occurred. The dangers of negative training have been observed in the field of aviation training, with a clear example seen through the crash of American Airlines flight 587. Analysis of the crash showed that simulator training with improper rudder force simulations resulted in a pilot who overreacted to real-world turbulence, shearing the tail off the aircraft due to extreme rudder inputs. This led to a crash that killed all 260 passengers and crew onboard the aircraft as well as five bystanders on the ground (Vidoli and Mundorff, 2012). The mismatch between simulator and real-world dynamics was found to be a leading cause of the accident, teaching an important lesson in the value of simulator fidelity and the dangers of negative training when utilizing a different raining environment (Myers et al., 2018).

However, our JIT subjects did not reach such elevated levels of difficulty during their limited inflight training, which may have helped avoid negative training. Note, though, that this effect may not have been observed if a higher difficulty level was used for postflight evaluation. To mitigate this in the future, the use of catch-trials, where the difficulty level is abruptly lowered periodically throughout training, could reorient subjects to performing easier presentations of the task. This may be particularly pertinent for manual control (i.e., piloting) skills to prevent overreaction and ensure that motor skill can adapt to a change in the magnitude of wind noise. The use of catch trials as a method to train adaptable motor skills has been previously implemented (Focke et al., 2013; Overduin et al., 2006), though caution should be taken in how frequently these trails are presented (Stockinger et al., 2014). In addition to the disconnect in difficulty level experienced in the Weekly training vs. the T2 Evaluation, it should also be reemphasized that the Weekly training was in VR (as was the JIT), while the T2 Evaluation was in the physical cockpit simulator. Extended training in VR may have also contributed to negative training for the evaluation in the physical cockpit simulator.

Subjective experience

At the T2 evaluation timepoint, adaptive JIT training shows the highest trends of flow, task importance, perceived skill, and perceived challenge-skill balance, though results failed to reach statistical significance. Participants who experienced no refresher training reported the highest perceived task difficulty, whereas weekly trained subjects reported the lowest. This trend is an expected outcome given the exposure to high levels of difficulty achieved in the weekly condition and the contrast when returning to a lower level of difficulty during evaluation. Despite repeated training at higher difficulties, the weekly condition reported lower average perceived skill than the JIT condition, and the lowest perceived task importance across all conditions. When skill gain occurs faster than changes in difficulty, boredom is likely to result (Engeser and Rheinberg, 2008). Previous studies have found that slight overload in demands, as opposed to true skill-demand balance, is optimal for the training-essential experience of concentration (Moneta and Csikszentmihalyi, 1999), though this preference may be strongly influence by personality traits of sensation seeking and action orientation (Baumann et al., 2016).

Divergent subjective experiences between JIT and weekly refresher training again trended in confidence reporting, where the highest average confidence of success reported before T2 evaluation tended to be reported by the JIT condition, the lowest in the weekly condition. While not significant, this may suggest that weekly training at such a high level of difficulty disrupted the self-efficacy and confidence of participants in that cohort, an effect that was mitigated by JIT training. Observed reciprocal relationships between self-efficacy and performance in academic settings support the idea that effective instructional design encourages confidence (Honicke et al., 2023). This improved confidence may have contributed to the performance demonstrated by the JIT cohort.

Implications for training design

JIT training is growing in popularity as a method for retaining critical skills while reducing the burden of time spent in refresher training. However, design of JIT training can be challenging, as it relies on prediction of how skill is expected to decay and how much or what kind of refresher training can mitigate that decay (Barshi and Dempsey, 2016). Our study demonstrates the adaptive JIT training can effectively mitigate skill decay and maintain proficiency, even when conducted in a VR simulation environment.

JIT paradigms of refresher training are particularly enticing for spaceflight applications, allowing crew focus on other onboard activities and utilize JIT training immediately before performance rather than devoting hours each week during flight to refresh critical skill. This has been explored in the context of on-orbit ultrasound examination skill on ISS, where JIT training was found to support crewmembers in obtaining clinically acceptable images of anatomy (Fo et al., 2005). However, this study utilized a ground-based expert to facilitate performance by watching downlinked videos and providing verbal instructions. This type of facilitation will not be possible with the communication delays of a Mars mission. Autonomous facilitation, such as that provided by adaptive difficulty training frameworks, are essential for JIT applications in LDEM. Advanced algorithms can be used to identify areas of weakness to target during JIT sessions. The results of this study support an alternative to ground-operator facilitated JIT training using adaptive difficulty. This demonstrates that VR technology can be used to create operational training scenarios with adaptive difficulty that can be initialized at any level relative to a trainee’s current skill.

Limitations

The use of a personalized adaptive algorithm, while the intent of the investigation, creates a between-subjects variability in training difficulty that is challenging to account for in results. Future investigations can combat this potential source of variance through larger sample sizes, or by the inclusions of a non-adaptive weekly or JIT training condition as an active control. Given the duration of this between subject training study, this study had a small pool of participants that were primarily recruited from college student populations, resulting in a younger age demographic that may influence how results translate to professional populations. We note that the JIT and weekly training groups differed both in terms of the total quantity of training (10 vs. 50 trials), as well as they cadence of training (none for 4 weeks and then 1 training session vs. 5 weekly sessions). Future work could disentangle these factors by assessing matched group of quantity vs. cadence (e.g., 10 trials over 5 weeks or 50 trials in a single session). Finally, while a consistent grading schema was applied to all conditions, the definition of acceptable performance on this physical, simulated spaceflight task was defined by the authors. Future work should evaluate how levels of retention on this type of task may correlate to real-world performance outcomes. Exploring the relationship between simulator training and real-world task performance will be a critical requirement to enable future development of training systems for high consequence industries.

The results of this study demonstrate that inflight training maintains critical skill and mitigates the decay of skill that occurs over periods of disuse. However, negative training at difficulty levels higher than performance requirements may impact performance. JIT training provided a robust retention of skill and a significant mitigation of skill decay as compared to subjects with no refresher training intervention. Together with the lower operational time burdens that it provides, JIT training is a promising method for the maintenance of critical work-related skills.

Future work exists in exploring more advanced algorithms, such as machine learning, which can be used to create a JIT training scenario following complex identification of gaps in skill. This could further amplify our results on how a trainee is targeted with refresher training without relying on human operators for facilitation. Within the human spaceflight domain, future work also exists in exploring the retention of skill over a longer duration between pre- and post-flight evaluation, as missions to Mars may create gaps in the range of 6–8 months, and where the impacts of physiological deconditioning that occur during spaceflight may also increase the rate of skill decay (Barshi and Dempsey, 2016; Heuer et al., 2003; Bloomberg et al., 2015; Stahn and Kühn, 2021). In sum, this research was the first to explore adaptive JIT training for operational training paradigms and found it to be successful in mitigating skill decay for simulation environments.