Towards automated assessment of communication skills in virtual reality: an exploratory feasibility study

1 Introduction

Virtual reality has matured into a practical medium for standardized social simulations that preserve contextual richness while maintaining experimental control. This combination is particularly valuable for assessment: VR can elicit job-relevant behavior in reproducible scenarios, while head-mounted sensing supports continuous capture of process data that is hard to obtain in conventional role-plays or interviews.

Professional communication is a key driver of how organizations function. It underpins coordination in teams, leadership effectiveness, and relationships with clients and stakeholders (Belasen, 1999). Yet, the tools most often used to assess communication skills—self-report questionnaires, structured interviews, and human ratings in role-play exercises—tend to lack ecological validity, are vulnerable to social desirability, and require considerable expert time. There is a growing consensus that more standardized and technology-supported procedures are needed, particularly in high-stakes contexts such as recruitment and professional training.

VR offers a middle ground: it allows researchers and practitioners to standardize the context and stimuli while maintaining a high degree of contextual and social realism. In social and communication domains, the concept of social presence, the feeling of being with another person, has been highlighted as a key prerequisite for using VR in the study of social behavior (Sterna and Zibrek, 2021). When this condition is met, VR has been shown to effectively support interventions in public speaking, clinical communication and job-interview skills (Riches et al., 2021; Kron et al., 2017).

1.1 Professional communication competence as a multidimensional construct

Professional communication competence has been broadly conceptualized as the ability to communicate in ways that are both effective and appropriate across interpersonal contexts (Spitzberg and Cupach, 1984; Rubin and Martin, 1994). In organizational settings, this competence cannot be reduced to the transmission of information. Professionals must explain ideas clearly, attend to others, manage disagreement, adapt their behavior to changing social demands, and maintain communicative control under pressure. For this reason, professional communication should be understood as a multidimensional construct that integrates verbal, non-verbal, paraverbal, and socioemotional components. This view is consistent with the present study, which focuses on seven communication-related competences: verbal communication, non-verbal communication, paraverbal communication, empathy, assertiveness, active listening, and emotional self-regulation.

At the most observable level, communication competence involves what is said, how it is delivered, and how the body supports the interaction. Verbal communication refers to the clarity, structure, relevance, and adequacy of spoken content. However, verbal content is interpreted together with non-verbal cues such as eye contact, gaze allocation, facial expression, posture, and gesture, which can shape the emotional and relational meaning of an interaction (Gunaratne et al., 2023; Shafique et al., 2023; Kolemba, 2015). Eye gaze is particularly relevant because it allows individuals to gather information about others’ attention and emotional state while also signalling involvement, responsiveness, and interpersonal orientation (Cañigueral and Hamilton, 2019; Jongerius et al., 2020). Paraverbal communication captures how the voice conveys meaning through tone, pitch, volume, rhythm, tempo, pauses, and vocal stability. These prosodic cues can modify the perceived intention of a message and influence listeners’ emotional responses, trust, and interpretation of the speaker’s attitude (Mitchell and Ross, 2013; Aguert, 2022; Klapprott et al., 2024).

Beyond these communication channels, professional interaction also depends on socioemotional competences. Empathy involves both the ability to understand another person’s emotional state and the capacity to resonate with that state (Genzer et al., 2023). In professional communication, empathy supports the recognition of social cues and facilitates more constructive responses to colleagues, clients, and stakeholders (Fuller et al., 2023). Active listening complements empathy by making attention and understanding visible through acknowledgement, responsiveness, and the integration of others’ viewpoints. Experimental evidence suggests that active listening behaviors increase perceived understanding and conversational satisfaction (Weger et al., 2014). Assertiveness, in turn, refers to the ability to express one’s views clearly and respectfully without becoming passive or aggressive. In applied settings, assertiveness has been associated with effective team communication and professional interaction, although its expression must remain sensitive to cultural and contextual norms (Susanto and Sofyan, 2024; Gutgeld-Dror et al., 2024).

Emotional self-regulation is a transversal dimension that affects all these components. Process models of emotion regulation describe how individuals influence the emotions they experience and how these emotions are expressed in behavior (Gross, 1998; Gross, 2015). In professional communication, this capacity becomes especially relevant in high-stakes interactions, where individuals must sustain verbal clarity, vocal control, attentional stability, and socially appropriate responses while facing pressure or evaluation. Effective regulation has been associated with better interpersonal functioning and more successful navigation of complex social situations (Webb et al., 2012; Koole et al., 2011; Koole, 2010). Thus, emotional self-regulation should not be treated as an isolated psychological trait, but as a regulatory layer that shapes how professionals communicate when interactions become demanding.

This multidimensional view has direct implications for VR-based assessment. Immersive simulations can elicit professional communication as situated behavior rather than as self-description. Gaze behavior can provide evidence of visual attention and non-verbal alignment; acoustic features can index paraverbal control and emotional activation; and speech transcripts can capture verbal structure, socio-affective language, and perspective-taking. This logic is consistent with social signal processing and multimodal human–computer interaction approaches, which treat non-verbal, paraverbal, and linguistic behaviors as interpretable signals of social interaction (Jaimes and Sebe, 2007; Vinciarelli et al., 2009). In the present study, VR scenarios are therefore used to observe how participants communicate across standardized but socially meaningful workplace situations, linking theoretical dimensions of professional communication competence to behavioral evidence captured during immersive interaction.

1.2 Stealth assessment and evidence-centered design in VR

The multidimensional nature of professional communication creates a methodological challenge: these competences cannot be fully assessed through isolated questionnaire items or decontextualized performance tasks. They become visible when individuals interact with others, respond to social cues, manage uncertainty, and regulate their behavior under contextual demands. For this reason, the present study adopts a stealth assessment approach, in which assessment is embedded within meaningful activity and behavioral evidence is collected unobtrusively during task performance (Shute, 2011; Shute and Ventura, 2013).

This approach is aligned with evidence-centered design, which provides a structured framework for linking theoretical constructs, assessment tasks, and observable evidence (Mislevy et al., 2003; Mislevy, 2013). In the present study, the competency model is defined by the seven communication-related skills described above: verbal communication, non-verbal communication, paraverbal communication, empathy, assertiveness, active listening, and emotional self-regulation. The task model is implemented through three standardized VR workplace scenarios that elicit different communicative demands. The evidence model is operationalized through naturalistic behavioral markers extracted from participants’ interaction with the virtual environment, including eye-tracking metrics, paraverbal voice features, speech transcripts, and decision-making patterns.

Within this framework, VR is not merely a technological medium, but the context that makes behavioral assessment possible. The simulated workplace scenarios provide ecological conditions in which participants must communicate as professionals while facing comparable task demands.

Following an Evidence-Centered Design (ECD) approach, the assessment framework assumes an explicit chain of inference linking constructs, task situations, observable behavioral evidence, and score interpretation. The present study focuses primarily on establishing the feasibility of this chain within immersive VR environments and should therefore be interpreted as an initial validation step rather than a complete validation program.

The central assumption is that these behavioral markers contain information about participants’ underlying communication competence. If VR scenarios successfully elicit relevant professional behaviors, then multimodal features extracted from those interactions should support the stratification of participants into low, medium, and high competence levels. This logic directly motivates the hypotheses of the study, which examine whether naturalistic behavioral markers captured in VR can provide meaningful evidence for automated classification across different dimensions of professional communication competence.

The objective of the present study is therefore to evaluate the feasibility of this assessment framework and its automated scoring approach, rather than to establish readiness for operational use in high-stakes professional assessment contexts.

2 Materials and methods

2.1 Participants and sample

The study followed a within-subject design in which all participants completed the same VR module, Impactful Communication. Adults with real work experience were recruited to ensure that the simulated situations could be interpreted against a genuine organizational background.

Inclusion criteria required participants to be at least 23 years old, resident in Spain, and to have accumulated a minimum of 2 years of professional experience. Exclusion criteria targeted safety and feasibility in VR, including a history of severe dizziness, vertigo or epilepsy, significant visual problems requiring surgery, and any condition that could interfere with correct use of the headset. Emotional health and ongoing treatments were screened to avoid undue distress during the high-stakes scenes.

Ninety-six participants met eligibility criteria and completed the VR module (51 men, 45 women). This number refers to module completers. The analytic sample for automated scoring was defined separately for each outcome because each model required a valid criterion label and a complete usable set of the corresponding multimodal predictors. After outcome-specific label construction and complete-case filtering, the available analytic samples were N = 93 for TECA empathy, N = 93 for RATHUS assertiveness, N = 95 for ERQ-RC, N = 95 for ERQ-SE, and N = 92 for the peer/expert video-rating outcomes. Therefore, all class distributions, majority-class baselines, and machine-learning results are reported using the valid N for the corresponding outcome rather than a single harmonized N across all analyses. Participants represented a variety of sectors and roles, reflecting the tool’s intended broad applicability. The study involved human participants and was reviewed and approved by the Ethics Committee for Research of the Universitat Politècnica de València (UPV; favorable opinion issued on 24 April 2024; reference: P04_24-04-2024). All participants provided written informed consent prior to participation.

2.2 Communication competence assessment scales

The analysis focused on seven communication-related skills targeted in the Impactful Communication module: verbal communication, non-verbal communication, paraverbal communication, empathy, assertiveness, active listening, and emotional self-regulation. Each competence was defined using established theoretical models and operationalized through a combination of psychometric instruments and expert ratings.

2.2.1 Empathy

Empathy was assessed with the Test de Empatía Cognitiva y Afectiva (TECA) (Lemos et al., 2022), which provides standardized indices of cognitive and affective empathy. These scores served as the reference labels for the empathy classifier.

2.2.2 Assertiveness

Assertiveness was measured using the Rathus Assertiveness Schedule (RATHUS) (Madrigal, 2009), quantifying the degree to which individuals express themselves clearly and confidently without being passive or aggressive.

2.2.3 Emotional self-regulation

Participants completed the Emotion Regulation Questionnaire (ERQ) (Orozco-Vargas et al., 2021), which assesses two major regulatory tendencies—cognitive reappraisal and expressive suppression. ERQ scores were used to generate two alternative operationalizations of emotional self-regulation included in the modeling framework.

It is important to note that these questionnaire-derived labels were used as criterion labels reflecting self-reported dispositional tendencies, rather than as direct observations of enacted behavior during the VR interaction. Therefore, for empathy, assertiveness, and emotional self-regulation, the models should be interpreted as predicting questionnaire-defined profiles from VR behavioral traces, not as directly measuring the full behavioral expression of these constructs within a single interactional context.

Competences related to observable communication performance (verbal communication, non-verbal communication, paraverbal communication, and active listening) required evaluation of behavior rather than self-report alone. Participants’ video-recorded VR performances and the standardized self-presentation video were reviewed with structured rubrics. However, the independent rater-level records needed to compute inter-rater reliability were not retained in a form that supports retrospective reliability estimation. Consequently, these expert/peer-derived labels are treated in the present manuscript as exploratory criterion labels rather than as reliability-supported ground truth.

This multimethod strategy provided preliminary labels from both validated self-report instruments and behaviorally anchored ratings. Nevertheless, these sources are not psychometrically equivalent. The questionnaire-derived labels reflect self-reported dispositional tendencies, whereas the expert/peer labels reflect aggregated judgements of observed performance. The revised interpretation therefore avoids treating all labels as equally validated criterion measures.

2.3 Virtual environment description

The Impactful Communication module was implemented as a standalone Unity-based immersive VR application delivered through a Meta Quest Pro head-mounted display. The module was experienced from a participant-view perspective: participants remained at predefined viewpoints within each scenario, did not freely navigate through the room, and could orient their head and gaze freely while the headset recorded head pose and eye tracking. Task interaction was implemented through controller-based selections, spoken responses, scripted dialogue prompts, and scenario event triggers. The system continuously recorded gaze data, speech audio, derived paraverbal features, automatic transcripts, and Unity event logs.

The module included three connected workplace scenarios presented in a fixed order: project meeting, one-to-one debriefing, and executive presentation. To remove ambiguity about the visual media, the first two scenarios were computer-generated 3D Unity scenes populated by scripted virtual agents, whereas the executive presentation used a static 360° panoramic boardroom scene embedded within the Unity application. The people visible in the panoramic boardroom were static, non-interactive human figures within the visual scene; they were not live participants, animated 3D agents, video recordings, or contingent interactive audience elements. In that scenario, interactivity came from the Unity layer: presentation prompts, timed phases, predefined audience-related AOIs, speech capture, gaze recording, and event logging.

Within VR, participants progressed through three interconnected workplace scenarios that mirror common leadership and communication challenges:

2.3.1 Project meeting

Participants interacted with four scripted virtual team members displaying different interpersonal styles and attitudes toward the project. This scenario was implemented as a computer-generated 3D Unity meeting environment with virtual agents, structured dialogue events, and response options. It elicited behaviors related to empathy, assertiveness, active listening, and non-verbal communication. The participant-view environment is shown in Figure 1.

2.3.2 One-to-one debriefing

Participants held a private conversation with a scripted virtual colleague who expressed concerns about misunderstandings and team tension. This scenario was implemented as a computer-generated 3D Unity office/debriefing environment and emphasized empathic listening, perspective-taking, and emotional attunement. The participant-view environment is shown in Figure 2.

2.3.3 Executive presentation

Participants delivered a formal presentation under time pressure and evaluative scrutiny. This scenario was implemented as a static 360° panoramic boardroom scene embedded within the Unity VR application. The visible audience functioned as a non-interactive visual and social-evaluative context, while the application recorded presentation speech, gaze allocation to predefined audience/room AOIs, and event markers for the presentation phases. It placed demands on paraverbal control, non-verbal alignment, audience scanning, and emotional self-regulation. The participant-view environment is shown in Figure 3.

The scenarios followed fixed narrative backbones with structured interaction points. They allowed limited variability in participant response options and speech delivery while preserving the same scenario order, task demands, visual layout, and event structure across individuals. The overall module duration was approximately 20–25 min.

To support reproducibility of the VR setup we added an operational scenario walkthrough in Supplementary Appendix S1.

2.3.4 Hardware and runtime

The simulation was delivered on a Meta Quest Pro head-mounted display with integrated infrared eye-tracking sensors. The application ran as a standalone VR experience, with interactions mediated through head pose, controller inputs, scripted prompts, and speech.

2.3.5 Integrated eye tracking

Prior to the scenarios, participants completed an in-headset calibration procedure. Gaze was recorded at the device-native sampling rate and time-stamped together with head pose. Areas of interest (AOIs) were defined for each scenario (e.g., interlocutor faces, team members, audience regions, and presentation-relevant artefacts) to derive interpretable gaze indicators such as AOI dwell time/proportion, contiguous AOI dwell episodes, and switching patterns. The retained analysis did not rely on a separate velocity- or dispersion-based fixation classifier; gaze features are therefore described as AOI-based dwell and allocation metrics. Quality-control rules were applied during preprocessing to handle tracking loss and invalid samples. The same preprocessing and quality-control procedures were applied to all participants before feature extraction.

2.3.6 Audio capture and speech processing

Spoken responses were captured through the headset microphone. Acoustic descriptors (e.g., prosodic and voice-quality features) were extracted from the raw signal using the fixed frame-based windowing procedure reported in Supplementary Appendix S1. Speech was also transcribed automatically to obtain lexical features. All modalities were synchronized through a common timestamping scheme so that features could be computed per scenario and aligned to task events.

2.3.7 Virtual agents, audience elements, and standardization

In the project meeting and one-to-one debriefing scenarios, virtual characters followed a structured interaction script that ensured comparable task demands across participants while allowing naturalistic conversational flow. In the executive presentation scenario, the audience was a non-interactive visual stimulus used to create evaluative pressure and to define gaze AOIs; no audience member generated contingent verbal or non-verbal responses. Scenario events and dialogue/presentation prompts were used to elicit target communication behaviors under realistic workplace pressure while maintaining reproducibility for assessment.

2.4 Experimental procedure

After eligibility screening and consent, participants first completed the psychometric battery (TECA, RATHUS, ERQ) and a brief demographic questionnaire (age, gender, professional background).

Before entering VR, participants recorded the standardized self-presentation video described above (≥2 min), which was stored for later peer evaluation.

They subsequently received standardized instructions and an explanation of the VR equipment. Once the headset was fitted, a short in-VR tutorial familiarized participants with the virtual environment and interaction mechanics. The Meta Quest Pro eye-tracking module was calibrated at the beginning of the VR session. Participants then experienced the three scenarios in a fixed order (project meeting, one-to-one debriefing, executive presentation) in a single session.

Throughout the experience, gaze data, audio signals and speech were continuously recorded. At the end of the VR session, participants completed additional questions related to usability and simulator discomfort, and were debriefed about the aims of the study.

2.5 Data processing

Data were obtained from three sources:

• psychometric scores from the TECA, RATHUS and ERQ;
• behavioral data from the VR scenarios (dialogue choices and interaction events);
• implicit measures, including eye-tracking metrics and acoustic features of speech.

All raw signals were pre-processed and transformed into a multimodal feature space combining eye-tracking, paralinguistic audio descriptors and lexical features extracted from speech transcripts. These signals were mapped onto seven communication-related competences, all treated as three-level categorical outcomes (Low, Medium, High) derived from external psychometric instruments and expert ratings.

Preprocessing procedures were standardized across participants and modalities. Quality-control procedures included removal of invalid gaze samples, handling of short tracking losses, normalization of acoustic features, filtering of transcription artifacts, and synchronization of all modalities through a common timestamping framework. Feature extraction was performed using predefined rules applied consistently across all participants and scenarios. The computational definitions of valid gaze samples, AOI assignment, AOI dwell metrics, acoustic frame/window features, transcript cleaning and tokenization, lexical dictionaries, and event/scenario aggregation rules are provided in Supplementary Appendix S1.

2.5.1 Verbal communication

From automatic transcripts of the meeting, office conversation and each phase of the final presentation, we computed counts and proportions of first-person pronouns and self-references, frequency of emotion-related and socio-affective terms, and lexical richness indicators (type–token ratio, mean utterance length). Features were normalized by total word count and aggregated per scenario.

Non-verbal communication. Eye-tracking data provided AOI-based gaze allocation measures, including valid-sample dwell time/proportion and counts of contiguous AOI dwell episodes for each team member while they were speaking and while the participant was responding, as well as gaze distribution across different audience regions and specific stakeholders during the presentation. These metrics indexed non-verbal alignment with the speaker, inclusiveness of visual attention and audience scanning behavior.

Paraverbal communication. For each event-log-defined dialogue segment (interactions with Rosa, David, Anne, Victor; the one-to-one office conversation; and the five presentation phases) we extracted summary descriptors of pitch, amplitude and RMS energy, percentage of silent frames/pauses and speech rate using the acoustic definitions reported in Supplementary Appendix S1. These descriptors characterised prosodic style, vocal intensity, pausing patterns and temporal fluency.

2.5.2 Empathy

Empathy features combined the proportion of correctly answered questions about the internal emotional state of virtual characters after the meeting with gaze-based indicators reflecting preferential attention to distressed characters and paraverbal softening during empathic segments.

2.5.3 Assertiveness

Assertiveness features captured how participants expressed disagreement, set boundaries and defended their position during team interactions, using acoustic descriptors from utterances directed to key characters together with gaze metrics indexing direct eye contact while responding and attention to the speaker while listening.

2.5.4 Active listening

Active listening was represented mainly through gaze metrics (valid AOI dwell time/proportion to the current speaker across meeting turns and presentation phases) and transcript features reflecting back-channel cues and reflective language, as well as the balance between self-focused and other-focused references. The label distributions for empathy and assertiveness are shown in Figures 4, 5.

2.5.5 Emotional self-regulation

For both self-regulation targets (ERQ-SE and ERQ-RC), we used a shared pool of features capturing behavior under evaluative pressure: paraverbal descriptors during the later presentation phases, gaze dispersion across the room, and global counts of first-person pronouns as a coarse proxy for self-focus.

For each competence we defined a specific subset of features, resulting in 15 features for verbal communication, 25 for non-verbal communication, 10 for paraverbal communication, 7 for empathy, 7 for assertiveness, 6 for active listening and 17 for emotional self-regulation. The exact variable names, denominators, computational formulas, and event/scenario aggregation windows are provided in Supplementary Appendix S1.

The exact feature dictionary and aggregation level for each predictor are specified in Supplementary Appendix S1, including whether the feature was computed at dialogue-turn, scenario, presentation-phase, or whole-session level before construction of the competence-specific feature matrix.

2.6 Statistical analysis

Analyses were conducted using outcome-specific analytic samples. For each criterion outcome, participants were included only if they had a valid Low/Medium/High label for that outcome and complete usable predictor values for the corresponding skill-specific feature set. We did not force a single complete-case sample across all outcomes, because doing so would have discarded valid information available for some scales. Consequently, the analytic N differs across outcomes and is reported explicitly in Table 1.

TABLE 1. Outcome label definitions, score ranges, and label distribution in the outcome-specific analytic datasets.

N refers to the valid analytic sample for each outcome after criterion-label construction and complete-case filtering of the corresponding multimodal predictor set. Ninety-six participants completed the VR, module, but analytic N varied by outcome because the availability of criterion labels, behavioral ratings, and usable multimodal features differed across scales/models.

Descriptive statistics were computed for all psychometric scores (means, standard deviations, medians and interquartile ranges). Continuous or quasi-continuous criterion scores were discretised into Low, Medium and High categories following the project scoring protocol, but the revised manuscript reports the exact cut-offs, class counts, and empirical majority-class baselines for transparency. The balanced 0.33 chance benchmark is retained only as a conceptual reference for a perfectly balanced three-class problem; it is not used as the sole benchmark for model interpretation because several observed class distributions were imbalanced.

The discretisation step created very small middle categories for some outcomes, particularly assertiveness (RATHUS; Medium n = 4) and emotional self-regulation based on ERQ-RC (Medium n = 7). These models are therefore interpreted with substantial caution, because overall accuracy may obscure failure to discriminate minority classes.

For expert/peer ratings, inter-rater reliability cannot be reported from the retained analytic dataset. Aggregated scores are insufficient for estimating agreement among independent raters, and no post hoc reconstructed rating matrix was used as evidence. Future validation studies should retain the full rater-by-participant matrix and report intraclass correlation coefficients for continuous rubric scores, confidence intervals, and category-level agreement indices after discretisation.

2.7 Machine learning and automated scoring

Our modeling objective was to examine whether automated scoring algorithms could extract preliminary signal about communication-related criterion labels from unobtrusively captured behavioral traces during the VR simulations. In line with the evidence-centered design logic that underpins stealth assessment, the VR scenarios served as the task model, the targeted skills represented the competency model, and the multimodal feature sets plus supervised classifiers operationalized an exploratory evidence model (Mislevy et al., 2003; Mislevy, 2013; Shute, 2011). The analysis should therefore be read as a proof-of-concept feasibility study, not as validation of an operational assessment system.

Outcome definitions (criterion labels). Each competency was modeled as a three-level outcome (Low, Medium, High), using external criterion labels rather than assuming a single homogeneous ground truth. Because the availability of criterion labels and usable feature streams differed by outcome, each classifier was trained and evaluated on the valid outcome-specific analytic sample reported in Table 1. For empathy, assertiveness, and emotional self-regulation, levels were derived from validated self-report questionnaires (TECA, RATHUS, ERQ) using range-based cutoffs (see Section 2.6). For behaviorally anchored outcomes (verbal communication, non-verbal communication, paraverbal communication, and active listening), levels were based on aggregated ratings of video-recorded performance. Because the independent rater-level data required to estimate inter-rater reliability were not retained, these behaviorally rated labels are treated as exploratory criterion labels only. The different label sources are theoretically related to the target competences, but they are not psychometrically equivalent and should not be interpreted as a single validated ground truth.

2.7.1 Predictor sets

Predictor variables consisted exclusively of multimodal behavioral features extracted from the VR session (Section 2.5), keeping a clear separation between model inputs and the criterion measures used to define competence levels. For each competency, we defined a skill-specific feature set a priori, grounded in theory and in the behavioral affordances of each scenario, to reduce researcher degrees of freedom and enhance interpretability. Across skills, predictors combined eye-tracking indicators of visual attention allocation, acoustic-prosodic markers of speech delivery, and linguistic indicators extracted from automatic transcripts.

Model training and evaluation. Each competency was treated as a three-class classification problem. For each skill, two predefined supervised classifiers were fitted to the skill-specific feature matrix: an XGBoost multiclass classifier and a Gaussian Process Classifier (GPC). For reproducibility, the resampling design was fixed before model comparison. The dataset was repeatedly split into 80% training and 20% test partitions across 48 iterations using independent random seeds; the seed used for each split was propagated to the stochastic components of the pipeline. SMOTE was fitted only on the training partition in each iteration and was never applied to the test partition. The 80/20 train-test splits were generated within each outcome-specific analytic dataset; no participant with a missing label or missing required predictors for that specific outcome was included in that outcome’s model. All preprocessing steps whose parameters can be estimated from data, including feature scaling if used in the final analysis script, were fitted on the training data only and then applied to the corresponding test data. Participants with a missing outcome label or missing values in the skill-specific predictor set were excluded for that particular model; no test-set imputation was used. For each model and skill, we retained the mean test accuracy and standard deviation across the 48 repetitions. Because fold-wise predictions and confusion matrices were not retained, balanced accuracy, macro-F1, class-specific precision/recall, and calibration indices cannot be computed retrospectively. Accordingly, accuracy is reported as a descriptive internal-resampling estimate and compared with empirical majority-class baselines rather than being treated as sufficient evidence of model quality.

Classifier configuration and reproducibility reporting. Hyperparameter optimization was not treated as a separate model-selection stage. The study compared two predefined classifier families under the same repeated train-test framework. The XGBoost classifier used a fixed multiclass soft-probability configuration, and the GPC used a fixed radial-basis-function kernel configuration; the exact classifier settings, kernel definition, SMOTE parameters, random-seed schedule, feature scaling/normalization rules, and software/package versions are now reported in Supplementary Appendix S1. Consequently, the reported results should be interpreted as exploratory estimates of scoring feasibility rather than fully optimized performance estimates.

Model comparison. The classifier with the higher mean accuracy and lower variability is reported as the selected model only for exploratory comparison. We no longer describe these models as operational scoring models, and we make no claim that the reported accuracy values establish deployment readiness.

Therefore, having a set of specific features for each skill (Table 2), we present the total number of features that were used for training the ML model in each skill.

TABLE 2. Number of skill-specific predictors used for multimodal automated scoring. Exact feature names and computational definitions are provided in Supplementary Appendix S1.

To facilitate reproducibility, Supplementary Appendix S1 has been rewritten as an operational reproducibility protocol rather than a checklist. It now reports the acquisition streams, timestamp synchronization and segmentation rules, modality-specific preprocessing, computational feature definitions and aggregation formulas, skill-specific feature engineering, missing-data handling, train/test split design, SMOTE settings, classifier hyperparameters and kernels, and software environment used for the exploratory ML analyses.

3 Results

3.1 Validity argument and evidence

Following contemporary personnel assessment standards, we structure the validity argument for VR-based automated scoring around multiple sources of evidence: (a) content and job relevance of the simulation tasks, (b) reliability and internal structure of criterion measures, (c) relations to other variables, and (d) consequences and fairness checks appropriate for high-stakes use. The present study provides preliminary evidence for content relevance and technical feasibility. It does not establish full construct validity, criterion reliability for expert/peer-rated outcomes, predictive validity, fairness across demographic groups, or readiness for operational use.

The Impactful Communication module was designed to sample behavior in situations that resemble common leadership communication demands. Participants enact the role of a team leader coordinating a product launch and encounter three standardized scenarios (project meeting, one-to-one debriefing, executive presentation). Table 3 maps each scenario to its VR implementation/media type, primary task demands, targeted communication-related KSAOs, and behavioral evidence sources.

TABLE 3. Mapping between VR scenarios, VR implementation/media type, primary task demands, targeted KSAOs, and evidence captured by the system.

For competencies anchored in observable performance, criterion labels were based on behaviorally anchored ratings of video-recorded performances. However, the independent rater-level data required to calculate inter-rater reliability were not retained in the analytic dataset. Therefore, the revised manuscript does not claim that these labels have demonstrated inter-rater reliability. They are retained only as exploratory labels for proof-of-concept modeling, and this limitation is reflected in the Results, Discussion, and Limitations sections.

Because AI-supported assessment in employment contexts is a potential high-risk use case, subgroup fairness is treated as a required part of the validity argument. In the present study, subgroup-level model performance analyses were not conducted and should not be inferred from the reported results. Future validation must link demographic variables to fold-wise predictions and report subgroup performance, differential error rates, and adverse-impact indicators before considering operational deployment.

However, subgroup-level fairness analyses were not conducted in the present study. Therefore, the current results should not be interpreted as evidence of equitable performance across demographic groups.

3.2 Automated scoring performance

Automated scoring was evaluated on the VR behavioral feature sets described in Section 2.5, with each competence treated as a three-class classification problem (Low, Medium, High). For every competence, we trained two supervised classifiers (XGBoost and Gaussian Process) on repeated 80/20 train-test splits, with SMOTE applied only to training data. The analysis provides internal descriptive performance estimates; it does not provide a complete assessment of model quality.

Table 4 reports mean test accuracy (± SD across splits), the higher-performing classifier for each outcome, and the empirical majority-class baseline derived from the observed label distribution. Because several outcomes were imbalanced, especially RATHUS assertiveness and ERQ-RC emotional self-regulation, the balanced 0.33 benchmark is insufficient as a stand-alone comparator.

TABLE 4. Exploratory internal accuracy estimates, empirical majority-class baselines, and revised interpretation using outcome-specific analytic samples.

Accuracy estimates and majority-class baselines were computed within the corresponding outcome-specific analytic sample reported in Table 1. Therefore, baselines differ according to each outcome’s observed class distribution and valid N.

The highest observed accuracies should therefore not be interpreted as evidence that the models are ready for operational scoring. In the absence of retained confusion matrices, balanced accuracy, macro-F1, class-wise precision/recall, calibration, and external validation, the results support only a cautious feasibility interpretation.

3.3 System performance, sensing reliability and user experience

Beyond model performance, VR-based assessment requires that the immersive system reliably produces analyzable behavioral signals without disrupting the interaction. To support feasibility, the pipeline incorporated quality-control and preprocessing steps for each modality, including detection of invalid gaze samples, handling of short tracking losses, and normalization of acoustic features to reduce sensitivity to microphone distance and background noise. Automatic transcripts were reviewed for gross errors that could distort lexical indicators, and out-of-vocabulary artifacts were filtered. These preprocessing decisions form part of the measurement model and should be considered when interpreting automated scores derived from multimodal behavioral signals.

Formal ASR performance indicators (e.g., Word Error Rate) were not available and should be reported in future validation studies to better characterize the reliability of transcript-derived features.

Participants also provided post-experience feedback on usability and simulator discomfort. These measures were used as system-level evidence to verify that the assessment could be completed within a short, single-session protocol while maintaining acceptable comfort. In line with the study goals, these system indicators are treated as necessary conditions for interpretable behavioral inference rather than as primary outcomes.

4 Discussion

This study examined whether a standardized VR workplace simulation can generate behavioral evidence for exploratory automated scoring of professional communication-related constructs. The findings provide qualified support for feasibility: multimodal markers captured during immersive interaction contained signal that produced internal accuracy estimates above empirical majority-class baselines for several outcomes. However, the results do not establish model quality, construct validity, or readiness for operational assessment. The strength of the evidence depends on the construct being inferred, the scenario affordances used to elicit it, the modality from which indicators are extracted, the criterion label source, and the adequacy of reliability and validation evidence.

The main contribution of the work is therefore not that a classifier reached a high accuracy value in an internal resampling procedure. Rather, the study demonstrates an end-to-end feasibility logic in which professional communication competences are defined theoretically, elicited through job-relevant VR scenarios, represented through gaze, speech and transcript-derived features, and linked to external criterion labels through supervised models. This logic remains preliminary. The present findings do not establish transportability, predictive validity in real-world settings, fairness across demographic groups, or suitability for consequential personnel decisions.

4.1 Interpretation of findings in relation to the study hypotheses

The first expectation was that immersive workplace scenarios would elicit observable behaviors relevant to professional communication. This expectation was supported at the system and design level. Participants completed three standardized but socially meaningful situations—a project meeting, a one-to-one debriefing and an executive presentation—and the system produced analyzable behavioral traces across eye tracking, paraverbal speech features and automatic transcripts. This is important because the purpose of the VR module was not to reproduce communication competence as an abstract questionnaire score, but to create situations in which competence-relevant behavior could emerge under comparable task demands. The mapping between scenarios, targeted competences and evidence sources supports the content logic of the assessment.

The second expectation was that multimodal feature sets would support automated stratification of participants into low, medium and high levels of competence. The internal results are consistent with this expectation at a feasibility level, but not at a validation level. Selected-model accuracies were descriptively higher than empirical majority-class baselines for several outcomes. Nevertheless, because some categories were extremely small and class-sensitive metrics were not available, we cannot determine whether the models discriminated all competence levels adequately.

The third expectation was that model performance would differ across competences and algorithms. This was confirmed by the divergence between XGBoost and Gaussian Process Classifier results. XGBoost was selected for verbal communication, assertiveness and one emotional self-regulation target, whereas Gaussian Process models were more robust for non-verbal communication, paraverbal communication, empathy, active listening and the second emotional self-regulation target. This finding argues against a single universal scoring model for professional communication. Instead, it supports a construct-specific approach in which each competence requires its own feature set, validation evidence and model-selection criteria. This result supports a modular assessment architecture in which competence-specific models are integrated within a common assessment framework.

4.2 Competence-specific meaning of the automated scores

The high internal accuracy obtained for assertiveness must be interpreted with particular caution. The RATHUS-derived label distribution included only four participants in the middle category. Under this condition, overall accuracy can be high even when minority-class discrimination is poor. Therefore, the result is reported as an exploratory signal, not as evidence that assertiveness can be scored reliably or validly by the current model.

The emotional self-regulation results require the same caution. Performance was particularly high for the ERQ-RC operationalization, but this outcome also had a very small middle class. These results may indicate that some regulatory tendencies are reflected in vocal stability, pausing, gaze dispersion and self-focused language during evaluative pressure, but they cannot establish a validated behavioral measure of emotional self-regulation.

This same caution applies to empathy. In the present study, empathy labels were derived from TECA scores and therefore reflect self-reported cognitive and affective empathy tendencies. The automated empathy model should consequently be interpreted as estimating the association between VR behavioral markers and a questionnaire-defined empathy profile, rather than as a direct measure of empathic behavior in the simulated interaction.

The moderate performance observed for empathy and active listening should not be read as a failure of the approach. These competences are socially complex and often depend on subtle relational timing, interpretation of another person’s state, and the appropriateness of responses across turns. A short standardized VR task can elicit some of these behaviors, especially through gaze to distressed characters, acknowledgement language and vocal softening, but it cannot fully capture the broader interpersonal history and reciprocal adjustment that characterize empathy and listening in real work settings. The above-chance results suggest that the current indicators contain relevant signal; the moderate accuracy indicates that richer dialogue features, improved semantic analysis and more extensive behavioral sampling will likely be needed.

For verbal, non-verbal and paraverbal communication, the findings support the value of multimodal inference. Communication performance was not represented by a single channel. Verbal clarity was linked to transcript-derived indicators, non-verbal communication to gaze behavior and attentional distribution, and paraverbal communication to prosodic and temporal properties of speech. The fact that these models performed above chance reinforces the theoretical assumption that communication competence is distributed across what participants say, how they say it and how they orient visually to others. At the same time, the moderate range of these results shows that current feature engineering captures only part of the construct. Future versions should integrate richer discourse-level measures, more precise turn-taking features and, where ethically acceptable, additional non-verbal indicators such as gesture, posture and facial behavior.

4.3 Implications for VR stealth assessment and evidence-centered design

The findings strengthen the rationale for using VR as a medium for stealth assessment. In evidence-centered design terms, the competency model was defined by seven communication-related competences, the task model was instantiated through three workplace scenarios, and the evidence model was operationalized through multimodal behavioral features and external labels (Mislevy et al., 2003; Mislevy, 2013). This alignment is central. Without an explicit chain from construct to scenario demand to observable behavior, automated scoring risks becoming a black-box prediction exercise. The present study instead treats machine learning as the final step in a broader validity argument.

This also clarifies what is meant by stealth assessment in the present context. The aim is not hidden psychological profiling, but embedded performance evidence: participants are assessed while engaging in meaningful professional tasks rather than by responding only to direct self-descriptive items. This approach is consistent with the logic of stealth assessment in digital environments (Shute, 2011; Shute and Ventura, 2013) and with social signal processing approaches that interpret gaze, speech and linguistic behavior as indicators of social interaction (Jaimes and Sebe, 2007; Vinciarelli et al., 2009). In a VR setting, the key advantage is that these signals are captured under standardized but socially realistic conditions.

Compared with conventional role plays or assessment centers, VR offers a stronger degree of stimulus control and repeatability while preserving some of the interpersonal pressure that makes communication behavior visible. Compared with self-report instruments, it captures enacted behavior rather than self-perception alone. The present results therefore support the use of VR as a complementary assessment layer. It should not replace validated psychometric scales or trained human judgment at this stage; rather, it can provide additional process evidence that may enrich a multi-method assessment battery.

From a validation perspective, the present results should be interpreted as supporting the feasibility of the proposed ECD chain rather than establishing full construct validity. While the study provides evidence that the scenarios elicit competence-relevant behaviors and that these behaviors can be linked to external criterion labels through automated scoring, additional evidence will be required to support score interpretation and operational use in professional assessment contexts.

4.4 Methodological and operational considerations

Several methodological points follow directly from the results. First, repeated train-test splits helped reduce dependence on a single partition of a modest dataset, and SMOTE was applied only within the training data to avoid leakage into the test sets. These are appropriate choices for an early proof-of-concept study. However, repeated internal validation is not equivalent to external validation, and accuracy alone is insufficient under class imbalance. The current performance estimates should therefore be viewed as internal descriptive evidence of possible signal, not as definitive evidence of generalizability or model quality.

Second, the discretisation of continuous scales into Low/Medium/High categories needs stronger validation. In this dataset, some middle categories were extremely small. Future analyses should consider alternative scoring strategies, including continuous regression, ordinal models, predefined normative thresholds, or larger samples that produce stable class sizes. Future reporting should include balanced accuracy, macro-F1, class-wise precision and recall, confusion matrices, calibration analyses, and confidence intervals.

Third, the choice and reliability of criterion labels are central. Some labels were derived from validated self-report instruments, whereas others were based on expert or peer ratings of observed performance. The latter cannot currently be supported with retained inter-rater reliability statistics. Future studies should preserve item-level and rater-level data, estimate inter-rater reliability before aggregation, and predefine rules for resolving disagreement. Until such evidence is available, expert/peer-derived outcomes in the present study should be treated as exploratory.

4.5 Implications for professional assessment and responsible deployment

For professional assessment, the practical implication is clear: VR-based automated scoring may be useful as decision support, not as an autonomous gatekeeping mechanism. At the current stage, the system should be understood as a research-stage assessment framework demonstrating feasibility rather than deployment readiness. Evidence regarding generalizability across organizations, cultures, and professional contexts remains to be established. The system can provide standardized behavioral evidence that is difficult to obtain at scale through traditional assessment centers, but the present results do not justify unsupervised selection decisions. This position is consistent with professional standards emphasizing validity evidence, appropriate use, documentation and fairness monitoring (Society for Industrial and Organizational Psychology, 2018; International Organization for Standardization, 2020). It is also consistent with recent discussions of machine learning in personnel selection, where gains in scalability and prediction must be balanced against transparency, construct validity and applicant fairness (Hardy et al., 2023; Ramos-Villagrasa et al., 2022). An additional constraint concerns fairness evaluation. Although demographic variables were collected, the present study was not designed to support robust subgroup performance analyses across all competencies and outcome classes. Consequently, the current findings provide no evidence regarding demographic fairness and should not be interpreted as supporting deployment in high-stakes decision-making contexts.

The integration of pre-recorded video ratings as complementary evidence is relevant in this regard. Asynchronous video assessment is increasingly used in recruitment and training contexts, but it raises concerns about rater bias, applicant experience, standardization and the possible use of generative AI to script responses (Dunlop et al., 2025; Koutsoumpis et al., 2024; Canagasuriam and Lukacik, 2025). In the present study, video-based ratings were used as criterion evidence rather than as a standalone screening method. This is a defensible use because it supports triangulation, but it reinforces the need for clear rubrics, multiple raters, reliability checks and transparent documentation.

Finally, governance requirements are becoming more stringent. In the European context, employment-related AI applications are treated as high-risk under the EU Artificial Intelligence Act (European Parliament and Council of the European Union, 2024). Any future deployment of this system should therefore include model cards describing intended use, target population, input data, performance and limitations; data documentation covering sensor conditions, preprocessing and missing-data handling; version control for datasets, feature extraction and model parameters; subgroup fairness and adverse-impact audits; and human oversight procedures that allow review and contestability. In plain terms: the system may help assess communication, but it should not be allowed to make consequential decisions alone. The algorithm gets a seat at the table, not the chairmanship.

Overall, the findings support the feasibility of the proposed VR evidence-generation pipeline while setting clear boundaries around interpretation. VR scenarios can elicit communication-relevant behavior, and multimodal features extracted from those scenarios can be used in exploratory classifiers. However, the current evidence is insufficient for claims of validated automated scoring. The next phase should move from proof of concept to validation: larger samples, retained rater-level data, reliability analysis, class-sensitive model evaluation, independent test sets, fairness audits, and longitudinal evidence linking VR-derived scores to real-world professional outcomes.

5 Limitations and future directions

Several limitations should be acknowledged. First, the sample was restricted to professionals in a single country and cultural context. Communication norms differ across cultures, and the external validity of the models should be tested in more diverse populations and organizational settings. Second, although the scenarios cover central aspects of leadership communication, they are anchored in a specific industry narrative. Future work should extend the approach to different sectors, such as healthcare, technology or customer service, and to different roles, including frontline employees, project managers and executives, potentially with tailored scenario design but reusing the same modeling principles (Kreijns et al., 2024; Shafique et al., 2023).

One limitation concerns construct alignment across criterion labels. The study combines questionnaire-derived labels and expert/peer behavioral ratings, which are not conceptually equivalent. This limits the interpretation of some automated scores, especially for latent socioemotional constructs such as empathy and emotional self-regulation. Future validation should include behaviorally anchored ratings of these constructs within the VR scenarios, examine convergent and discriminant relationships between self-report, expert-rated behavior, and automated VR-derived indicators, and test whether these scores predict real-world professional communication outcomes.

A central limitation is that the independent rater-level data required to estimate inter-rater reliability for expert/peer-rated outcomes were not retained in the analytic dataset. As a result, inter-rater reliability cannot be reported and these labels should not be treated as reliability-supported ground truth. Future studies must retain the full rater-by-participant matrix and report agreement indices before aggregation.

The current evaluation is cross-sectional. An important next step is to examine longitudinal predictive validity, specifically whether competence levels inferred from VR behavior predict future job performance, team outcomes or leadership effectiveness in real organizations. Finally, while the pipeline already supports multimodal integration, future research can explore real-time adaptation of scenarios based on ongoing behavioral signals, creating more interactive and personalized simulations, as well as leveraging explainability techniques to provide more transparent feedback to users and HR professionals.

Another limitation concerns performance evaluation. Although repeated train-test splits provided stable estimates of mean accuracy, several outcomes exhibited substantial class imbalance, and some middle categories were extremely small. Consequently, accuracy may not reflect the ability of the models to discriminate among all competence levels. Future work should report balanced accuracy, macro-F1, class-wise precision and recall, confusion matrices, calibration metrics, confidence intervals, and external validation results.

A different limitation concerns fairness evaluation. No subgroup analyses were performed to examine whether model performance differed across demographic groups such as age or gender. As a result, the present study cannot support claims regarding fairness, absence of bias, or equitable model behavior. Future validation studies should include subgroup-level performance analyses and fairness audits before considering operational deployment.

Generative AI statement

The author(s) declared that generative AI was used in the creation of this manuscript. During the preparation of this work, the authors used ChatGPT for language editing and stylistic improvement of the manuscript. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2026.1872244/full#supplementary-material

Participant-view walkthrough of the three VR scenarios in the fixed order used in the study: project meeting, one-to-one debriefing, and executive presentation. The video documents the participant viewpoint, visual media type, scripted prompt structure, and interaction points; it does not include participant-identifiable data. Supplementary material at the source

Supplementary material Supplementary material at the source

Footnotes

Edited by: Abdul Waheed Siyal, Wuxi University, China

Reviewed by: Syed Azeem Inam, Sindh Madressatul Islam University, Pakistan

Mohamad Da’i, Nahdlatul Ulama Sunan Giri University, Indonesia

Muhamad Arif Mahdiannur, Surabaya State University, Indonesia