Original Research
Frontiers in Virtual Reality ·
Volume 7 - 2026 | https://doi.org/10.3389/frvir.2026.1801743
The design, gamification, and physiological evaluation of a VR social interaction simulation
- Pranav Parekh1*corresponding author
- Oluwafemi Richard Oyeleke1
- Stevens Institute of Technology, Department of Systems Engineering, Hoboken, NJ, United States
Abstract
Communication skills are essential characteristics required to thrive in a fast-paced and ever-growing society that we currently live in. The ability to socialize in both informal and formal settings is important for the all-around development of individuals. The use of VR simulations for training has provided a novel avenue for individuals to learn such soft skills. In this work, we present a VR-based interaction system designed to simulate four structured social scenarios. The primary objective of this study is to evaluate whether the proposed simulation can elicit measurable physiological responses in users during controlled social interaction tasks. We analyze the participants’ autonomic responses using heart rate variability (HRV) as an index of physiological arousal associated with stress-related responses. Initially, we discuss the various anxiety disorders and expand on social anxiety disorder (SAD). SAD is used here as a theoretical motivation for designing socially evaluative VR environments rather than as a clinical target of evaluation. Thereafter, we study the connection between social skills training (SST) and SAD, as well as the possibility of imparting training as exposure therapy. It is key to note that the present work does not directly evaluate training outcomes or behavioral skill acquisition. Instead, we focus on the role of VR in supporting controlled exposure to social situations rather than demonstrating improvements in confidence or social skills. In addition, we begin by creating a scene through four key scenarios by designing their avatars and adding pre-scripted conversations. Finally, we evaluated the simulation on two key aspects, namely, affective fidelity and construct validity. We use graphical representations to infer that the simulation elicits differentiated patterns of HRV across users and scenarios, and this varies appropriately from one user to another. We therefore infer that the simulation demonstrates its ability to induce and measure stress in structured social interaction contexts, rather than constituting or validating a social skills training intervention.
Keywords
1 Introduction
Anxiety is a natural emotion characterized by feelings of tension, fear, and worry. This often results in physiological changes such as increased blood pressure. Occasional anxiety is a necessary and normal part of life. Anxiety disorders involve more than temporary worry and can lead to depression. Anxiety typically does not resolve on its own and may worsen over time without intervention (Barlow, 2004). It can affect day-to-day activities and the general wellbeing of a person. There are several types of anxiety disorders, including Generalized Anxiety Disorder (GAD) (Stein and Sareen, 2015), Panic Disorder (Roy-Byrne et al., 2006), Social Anxiety Disorder (SAD) (Stein and Stein, 2008), and phobia-related disorders. People with SAD have a general fear of social situations. They are deeply affected by the thoughts and judgments of other people. Common symptoms associated with SAD may include sweating, trembling, elevated heart rate, gastrointestinal discomfort, rigid body posture, and difficulty maintaining eye contact during social interactions. Prior research suggests that these responses are frequently associated with heightened self-consciousness and fear of negative evaluation. Such socially evaluative situations are therefore useful contexts for studying physiological stress responses during social interaction.
Cognitive Behavioral Therapy (CBT) is a form of psychological treatment that has been effective for a variety of mental health problems, such as depression (López-López et al., 2019), anxiety disorders (James et al., 2020), alcohol addiction, drug use problems, eating disorders, and any other form of mental illness. Efforts are made to alter thinking and behavioral patterns to improve daily functioning and quality of life. These strategies include (Beck, 2011):
- Recognizing lapses in thinking that create problems and evaluating how to avoid them.
- Understanding the behavior and motivation of others to avoid negative assumptions.
- Facing your fears to develop a sense of confidence.
- Using problem-solving and thoughtful decision-making during difficult situations.
- Role-playing or rehearsing specific situations to develop effective coping strategies.
The simulations proposed in the following manuscript are also based on the third and fifth CBT strategies mentioned above, which serve as the conceptual basis for the design of the social interaction scenarios.
Social Exposure Therapy (SET) is a fundamental behavioral technique within the CBT framework. It emphasizes the gradual exposure of a person in social situations to reduce avoidance behaviors and increase familiarity with social interactions (Beidel et al., 2014). This is because social anxiety typically arises from overthinking, fear of judgment, and self-consciousness in social situations. In the context of SET, Social Skills Training (SST) is a key form of rehearsal commonly used to help individuals practice social interactions in a structured environment. This increase in confidence alleviates social anxiety by making social interactions feel natural to the user (Olivares-Olivares et al., 2019). In the long term, SST, when rendered in the form of exposure therapy, keeps SAD away by improving self-efficacy and confidence, and reducing negative thoughts stemming from the fear of embarrassment. The concepts of SET and SST therefore provide the theoretical motivation for the design of the proposed VR scenarios. Accordingly, the present work focuses on developing and evaluating a VR-based social interaction simulation that exposes users to socially relevant situations and measures the associated physiological stress responses. The skills that we would like to improve are initiating conversations, active listening, and conflict resolution, among others.
While social anxiety is fundamentally a psychological and subjective condition, anxiety-provoking social situations are known to elicit measurable physiological stress responses through activation of the autonomic nervous system. Prior research has demonstrated that individuals with elevated social anxiety exhibit increased physiological stress, most notably reduced heart rate variability (HRV) when exposed to socially evaluative or performance-based contexts (Thayer and Lane, 2000). Accordingly, this study employs HRV as an indirect physiological indicator of affective responses associated with social anxiety, rather than as a direct or standalone measure of anxiety itself.
In the following research, we leverage VR and simulation design to draw upon principles commonly employed in SET and SST interventions. This leads us to Virtual Reality Exposure Therapy (VRET), which is the use of VR simulations to immerse patients in a computer-generated environment that replicates scenarios systematically comprising their triggers. Through this, patients become more comfortable while experiencing this in the real world (Krijn et al., 2004). On the other hand, in-virtuo exposure is the use of VR simulations to improve our skills by practicing scenarios on the simulation. It differs from VRET because it can serve as a standalone training and is not a part of a structured therapy (Bouchard et al., 2011). The proposed system is conceptually inspired by both in-virtuo exposure and exposure-based approaches, as it provides users with repeated exposure to socially relevant interaction scenarios within a controlled virtual environment. However, the present study does not evaluate therapeutic outcomes or social skill acquisition directly, and instead focuses on the physiological responses elicited by these interactions. Other existing VR simulations focusing on SST comprise a single tone (like formal or informal) and provide training on a single competency (for example, public speaking). The proposed environment incorporates multiple social contexts that differ in social complexity, conversational demands, and perceived evaluative pressure. To conclude, we make the following contributions through this manuscript:
- Designing a VR-based social interaction simulation that presents a range of socially evaluative scenarios relevant to social skills training and social anxiety research.
- Incorporating gamification to make the simulation engaging, enjoyable, and goal-oriented (Nor et al., 2019).
- Evaluating the simulation through affective fidelity and construct validity using participant physiological responses.
- Using HRV as a physiological indicator of autonomic stress responses during the social interaction scenarios.
While prior VR simulations typically focus on single scenarios, our work introduces a multi-scenario design that spans both formal and informal social contexts. Each scenario is structured across five engagement levels to increase verbal participation progressively. Unlike much of the existing literature, our evaluation employs fidelity and validity measures to assess whether the simulation elicits authentic social responses from the user. This combination of scenario diversity, gamification concepts, and rigorous evaluation constitutes the main innovation of our system. We begin by performing a literature review on the existing VR simulations, gamification ideologies, and evaluation techniques used in this research project.
2 Literature review
VRET and in-virtuo exposure offer several advantages in therapy over traditional methods. Virtual environments can be designed based on real-life scenarios, but situational elements can be controlled. Therefore, VR allows researchers and therapists to manipulate certain scenarios within the simulation (Sl et al., 2016). By manipulating situational elements, therapists can tailor scenarios to suit the needs of the patients. This is the fundamental principle upon which exposure therapy and skills training are planned using a VR environment. Secondly, using VR allows individuals to engage in exposure therapy within the comfort and safety of controlled environments. Individuals can pause or stop the training anytime, granting them control over the process (Rothbaum et al., 1995). VR also allows the close monitoring of patient reactions to given scenarios in a simulation. Data based on physiological and behavioral markers such as heart rate, skin conductance, and movement patterns provide objective measures of the responses given by the patient. Through monitoring capabilities, the therapist gets a comprehensive understanding of the patient’s progress (Bouchard et al., 2017).
VR technology enables patients to experience simulated situations that are more intense than real life, potentially enhancing therapeutic outcomes. Using such environments, therapists can amplify the intensity of the simulation beyond what is commonly experienced in the real world. Simulations can also include highly immersive scenarios, which are useful when anxiety triggers are intangible (Grenier et al., 2015). For example, exposure therapy is better suited to using a VR simulation in cases of arachnophobia (fear of spiders) (Andersson et al., 2024), agoraphobia (fear of situations where escape might be difficult) (Yang et al., 2024), acrophobia (fear of heights) (Suyanto et al., 2017), and many others. Patients undergoing VRET have the advantage of replaying the simulation as many times as needed. Patients can revisit scenarios whenever necessary, allowing them to build resilience and confidence over time. The repetition contributes to the eventual habituation and consolidation of therapeutic gains. This characteristic is essential in cases of PTSD (Rizzo et al., 2013) and anxiety disorders (Gorini and Riva, 2008).
VR training applications are typically created to address specific tasks using a standardized procedure to achieve a specific training objective (Xie et al., 2021). They have been developed for various domains, including but not limited to safety training (Li et al., 2018), escape training (Li et al., 2017), education (Wang et al., 2018), and mental health improvement. Therefore, their implementations are task-oriented and distinct from one another. However, there have been attempts to generalize a pipeline that is useful while creating VR simulations (Lin et al., 2002). introduced VR training task planning and presented task-oriented training scenario models within the architecture of a VR training system. Overall, the creation pipeline of VR training was reduced to three key stages: scenario design, task analysis, and implementation. We extend these principles to the design of gamified VR social interaction simulations intended to expose users to a range of structured social scenarios. Furthermore, we explore the use of physiological measures to evaluate user responses across scenarios with varying social demands.
Several applications have been designed to achieve the goal of efficient SST within a simulation. Considerable research has used SST as an intermediate technique to achieve improvements in Autism Spectrum Disorder (ASD) (Ke et al., 2022), schizophrenia (Park et al., 2011), Social Anxiety Disorder (SAD), and so on. Improving social skills enhances the patient’s confidence and positive outlook, thereby serving as therapy in numerous scenarios. Social anxiety triggers can also be intangible, like overthinking, fear of judgment, internal beliefs, etc. SST is useful in helping patients feel confident from within, as they practice handling numerous social situations, thereby addressing internal social anxiety triggers. Currently, In vivo Exposure Therapy (iVET) is the most common treatment for SAD. This method involves gradually exposing the participant to anxious stimuli to improve the subject’s reaction to a given situation. The paper by (Kampmann et al., 2016) compares the iVET method to the VRET and concludes that using iVET is more efficient for subjects, especially since the effectiveness of VRET is highly dependent on the quality of the scenarios within the simulation (Caponnetto et al., 2021). points out that the iVET technique incurs higher costs and more time since it involves actual people within the simulation. Therefore, despite stronger outcomes reported in some iVET studies, we focus on VRET due to its greater accessibility, ease of deployment, and ability to provide repeatable social interactions within a controlled environment. Additionally, VR-based simulations allow the inclusion of diverse social settings with varying levels of interpersonal and cognitive demand, which may support broader assessment of social skills.
VR has emerged as a powerful medium for facilitating SST and treating SAD by providing safe and repeatable social environments. Early VR interventions primarily targeted performance-based anxiety, particularly public speaking. Studies such as (Anderson et al., 2013) used virtual audiences who expressed reactions, ranging from neutral to critical, to expose participants to feared responses. These simulations reduced avoidance behaviors by enabling users to confront their social fears. Further developments expanded to include job interviews and classroom participation to recreate complex social dynamics (Lindner et al., 2019). The study by (Ferreira et al., 2024) focuses entirely on VR for public speaking, where speakers are assessed on indicators like voice tone, head movements, and speech duration. On the other end of the spectrum, the study by (Pattiasina et al., 2025) focuses entirely on treating Social Anxiety Disorder (SAD) through a scoring engine built over the virtual social interactions (Kahlon, 2022) also incorporated gamification through reward mechanics to enhance user engagement. Despite these advances, current systems focus entirely on a singular scenario with a limited difficulty level. Despite these advances, many existing systems focus on a single social context or a limited range of interaction difficulty. In this work, we investigate a broader set of social interaction scenarios with varying social demands and examine whether these scenarios elicit differentiated physiological responses across participants.
In most cases, the importance of VR simulation design has exceeded the testing and validation of the simulation tool. However, the increasing use of VR has intensified research on key evaluation metrics, such as fidelity and validity. A successful VR environment enables the transfer of learning in the real world (Harris et al., 2020). Validity is the extent to which a simulation provides an accurate representation of its real equivalent. It is further categorized into face validity and construct validity (Bright et al., 2012), which we apply in this simulation as follows:
- Face validity: Refers to the design and technical capabilities of the simulation. It focuses on how the simulation appears, rather than the actual act of learning. We require a sufficient level of face validity within the simulation, but are more concerned with the functional aspects. We do not look to evaluate face validity since ultra-realism and graphics are secondary objectives for this research.
- Construct validity: This is very important to evaluate since it focuses on accurately representing the task within the simulation. A simulation with good construct validity should be sensitive to performance variations between and within individuals. Therefore, we plan to evaluate this metric by comparing the results of multiple users, users with varying degrees of social intelligence and confidence. If they exhibit contrasting reactions, we can deem the simulation effective in highlighting varying emotions in different users and consider it to have high construct validity.
On the other hand, fidelity is the extent to which a simulation recreates the real-world system in terms of the appearance, cognitions, and behavior it elicits from the user (Liu et al., 2008). This is divided into four categories, which are useful to this simulation as follows:
- Physical Fidelity: It is based on the realism provided by physical elements like laws of physics, level of functionality, lighting, etc. Since our simulation is relatively simple and usually within closed spaces with conversations, we are less concerned with evaluating the physical fidelity of the simulation.
- Psychological Fidelity: This is important to evaluate since social anxiety is often a psychological issue. It refers to how well the simulation elicits perceptive and cognitive tasks from the user.
- Affective Fidelity: For VRET simulations that treat anxiety disorders, high affective validity is essential since it requires the simulation to elicit emotional responses like stress, fear, or excitement. We plan to measure the user’s stress levels using a smartwatch that captures ECG when the user is in the simulation. We compare the readings between the baseline and simulated tasks to quantify the affective validity of the simulation.
- Ergonomic Fidelity: Ergonomic Fidelity promotes realistic movement and is necessary for movement-based simulations. Since the following simulation involves little to no movement, we do not need to measure ergonomic fidelity.
In the following manuscript, we design an effective SST simulation and evaluate it using affective fidelity and construct validity, which was introduced above.
3 Methodology
We divide the methodology into two sections: “Creating a Scene” and “Gamification of the Scene”. The scenario design section introduces four scenarios where the user is immersed to test for social anxiety. Gamification of a scenario involves incorporating a scenario flow and offering rewards upon completion of each stage.
3.1 Creating a scene
Although there are some good VR options like Psious (Digital Innovation and Transformation, 2008) for treating anxiety disorders, the options provided for social anxiety specifically are few and do not consist of a detailed conversation within each scenario. On the other hand, for this simulation, we focus on enhancing important aspects of social skills across various settings. Using the simulation, we aim to explore various scenarios, including the Grocery Store, a Fancy Restaurant, a Job Interview, and a Networking Event. Together, the four scenarios encompass both formal environments (Job Interview and Networking Event) and informal interactions (Grocery Store and Fancy Restaurant), ensuring comprehensive coverage of everyday social demands. Current SST simulations do not generalize from one context to another. We use a wider variety since each scenario trains a different category of social skill. This coverage makes the VR system feel relatable and practically aligned with real-life social demands, ensuring that the skills practiced in simulation translate meaningfully into everyday use. Different users also have their respective social triggers, which they can practice accordingly.
To ensure that the VR system targets meaningful aspects of social functioning, we select core competencies through SST and CBT literature, like (Beidel et al., 2014). We identify five guiding competencies to reflect patterns of adult social interaction: initiating conversations, group interactions, public speaking, dealing with authority figures, and receiving feedback. These competencies are not intended to serve as isolated categories. Rather, they operate as conceptual anchors that shape the design and intent of each scenario. In practice, a single scenario may naturally engage multiple competencies simultaneously. For example, a fancy restaurant may require both initiating conversations and managing group interactions, while a job interview may blend public speaking with responding to evaluative feedback. By mapping the different scenarios to these competency domains, we can create specific flow structures that progressively increase social risk, interpersonal complexity, and cognitive load. This approach keeps the system clinically informed yet flexible, allowing each scenario to provide layered and meaningful social training experiences.
While defining a scene, we pair a scenario context with a conversation module rather than a single fixed competency. Each conversation module is designed to emphasize a particular social skill (such as initiating conversations), but the interactions within it may naturally involve multiple overlapping competencies. This flexible structure allows each scene to remain realistic and behaviorally rich while still maintaining a clear instructional focus. Formally, we denote a scene as (scenario, module). For example, (4, 1) refers to the networking scenario paired with conversation module 1, which may center on initiating conversations but also incorporates elements of group interaction and receiving feedback.
3.1.1 The avatars
Although avatars and animations are specific to each scenario, we design them first and then add them to the scenario. We utilize the Advanced People Pack Asset to design characters for each scenario. By using this asset, we develop character prefabs that contain a pre-configured GameObject, including scripts, animations, colliders, and materials. Character prefabs are fundamental for creating virtual representations of individuals within simulations. Prefabs streamline the integration of diverse characters into the Unity scene, enabling us to populate the four distinct scenarios. The first step is to load the character prefab onto the scene so that it can be customized accordingly. We adjust the height, body proportions, and facial features to create diversity in avatars. The character prefab is depicted in Figure 1.
 Asset via [Unity Asset Store](https://assetstore.unity.com/) and are subject to the [Unity Asset Store](https://assetstore.unity.com/) EULA.](https://www.frontiersin.org/files/Articles/1801743/xml-images/frvir-07-1801743-g001.webp)
The asset provides a customizable system for your character to allow unique clothing and distinct character styles. Designing clothing and accessories for these characters requires meticulous attention to detail to ensure realism and context relevance within the simulated environment. We can modify textures, colors, and styles using the material and shader systems to align with the functional requirements pertinent to the research objectives. For example, we create a uniform for the workers within the grocery store. The uniform includes an apron and accessories like a name tag. They serve as the distinguishing factor between a shopper and a worker at the grocery store. We modify the shaders to select a cotton t-shirt and a polyester apron for material realism. A sample avatar is shown in Figure 2, along with the parameters that can be altered according to the scenario.
 Asset via [Unity Asset Store](https://assetstore.unity.com/) and are subject to the [Unity Asset Store](https://assetstore.unity.com/) EULA.](https://www.frontiersin.org/files/Articles/1801743/xml-images/frvir-07-1801743-g002.webp)
Character locomotion is crucial for conveying natural movement and behavior realism. The Asset is designed with Unity’s animation tools to implement walking cycles that reflect biomechanical principles and behavioral nuances of the simulated characters. Although locomotion is a secondary objective, we adjust parameters like stride length, gait speed, and posture dynamics to depict alternate walking styles, contributing to an immersive and realistic experience. The walking motion of an avatar is depicted in Figure 3. The practice of utilizing character prefabs, designing attire and accessories, and refining locomotion improves visual fidelity and enhances the authenticity of the simulation.
 Asset via [Unity Asset Store](https://assetstore.unity.com/) and are subject to the [Unity Asset Store](https://assetstore.unity.com/) EULA.](https://www.frontiersin.org/files/Articles/1801743/xml-images/frvir-07-1801743-g003.webp)
BlendShapes allows you to modify the mesh geometry of a 3D character by transitioning between different shapes. It is particularly effective for facial expressions, subtle body modifications, and lip-syncing. To enable BlendShapes, we import the FBX model into the Unity Assets folder. We select the character prefab in the scene and locate the Skinned Mesh Renderer component in the Inspector. We can adjust values from 0 to 100 to modify facial expressions and body morphs. We can adjust the values sequentially at different keyframes to create animations such as smiling, talking, and changing expressions. We change expressions dynamically by adding the animation clips using C# scripts. For example, when a buyer who does not find an item in the grocery store changes their expression from happiness to disappointment.
In our system, BlendShapes are not implemented as pre-authored animations. Instead, they operate through an Integrated Expression Aligned Dialogue Pipeline that links emotional intent directly to each line of the character’s script. Every dialogue segment is assigned a specific expression profile: neutral, polite, friendly, confusion, concern, disappointment, fear, disgust, anger, and authority. When a dialogue line is triggered during the simulation, its associated expression profile automatically activates a corresponding BlendShape configuration through the custom C# controller. This allows facial expressions to adjust dynamically and contextually, producing real-time emotional responses rather than rigid animations. To enhance realism further, jaw movement for spoken lines is driven by the amplitude of the character’s audio, enabling lightweight, real-time lip-sync. By coupling scenario design with conversation modules, this pipeline ensures that each avatar displays expressive behavior that is tightly aligned with the social demands of the interaction.
3.1.2 The scenarios
As mentioned above, we have four different scenarios to test the five key points of social communication. We create a dedicated scene folder for each scenario, which includes all the essential GameObjects and required avatar designs. A scene folder consists of multiple scenes that test different key points. In the following section, we will establish which scenario focuses on which key point, how they are created, and how they comprise the scene.
- Store Visit: It is created as a conversational, light-weight scenario with an informal tone and casual communication. We utilize the grocery store asset to create a supermarket scenario. We randomly add avatars to the scene, which fall into three categories: customers, cashiers, and workers. The functionality of the scene tests our avatar (one of the customers) to either initiate conversations or respond to the cashier or workers. For example, a scene may focus on searching for an item in the store, and if it is not found, the avatar must initiate a conversation with a worker. A billing scene may involve making conversation with the cashier. The cashiers and workers are identified by their aprons. Figure 4 depicts the creation of the supermarket simulation. We have added two characters: the cashier behind the counter and the customer walking towards the counter.
- Fancy Restaurant: It is also informal, but includes more instances of group conversations among friends and with the restaurant workers. We use the modern restaurant asset to design the scenario. The avatars are of two categories: the customers and the waiters. Our avatar is a customer who has arrived with a group of friends for a meal at the restaurant. He is tested on initiating a conversation with the waiter and placing the order for the entire group. Apart from that, the scene consists of a group interaction between our avatar and their friends.
- Job Interview: This scenario is designed to be entirely formal and clearly aligned with authority and feedback handling. We use the office room asset to build the job interview scenario. The avatars fall into two categories: the employee and the employer(s). Our avatar is the employee being interviewed for a job. We host a behavioral interview, which is a back-and-forth question-and-answer format between the two characters. Through this scenario, we aim to assess the user’s ability to communicate effectively with authoritative figures and respond to their questions. We also assess the user’s reactions when the employer provides feedback after the interview. Increasing the number of employers directly increases the difficulty of the scene.
- Networking event: The networking event is well-balanced with both formal and informal tones. It is built using the office room asset and is designed to evaluate all five communication competencies. The environment includes three types of avatars: students, company representatives, and external professionals. It places the user in the role of a student navigating a realistic networking session. Throughout this scenario, the user may be asked to introduce themselves or share their resume with recruiters, participate in debates or informal group discussions, pitch ideas to a small audience, interact with managers or senior employees, and handle criticism or rejection from company representatives. By combining all five skill categories into one cohesive experience, this networking scene functions as the most comprehensive and challenging scenario in the entire simulation.
 and [Advanced People Pack](https://assetstore.unity.com/packages/3d/characters/humanoids/humans/advanced-people-pack-2-170756) via [Unity Asset Store](https://assetstore.unity.com/) and are subject to the [Unity Asset Store](https://assetstore.unity.com/) EULA.](https://www.frontiersin.org/files/Articles/1801743/xml-images/frvir-07-1801743-g004.webp)
3.1.3 The conversations
The conversations for each scenario play a crucial role in making social situations feel authentic. We use a predefined script that appears at the bottom of the screen for the user to engage with the avatar. We select five distinct scripts focusing on each scenario to ensure the simulation feels realistic and coherent. We reference the following scripts for each scenario:
- Grocery Store: The Grocery Store conversations include conversations between the worker and customer, the cashier and customer, and between two customers. We use the script provided in (studocu, 2022) that includes all three instances.
- Fancy Restaurant: The conversations include calling and ordering food from the waiter, and an internal discussion on a topic with friends. Both varieties of instances are found on (thoughtco, 2024).
- Job Interview: We use a personal Q&A document that includes behavioral questions asked in an interview.
- Networking Event: Since the event includes numerous key points, we use scripts from two different websites (careerpotential, 2026; southwestern.edu, 2025). These scripts focus on general networking conversations, like two visitors conversing or a visitor speaking to an employer. We also include generic questions where the user is asked to pitch themselves within the simulation.
After accumulating the scripts for each scenario, we aggregate them into a CSV file. Sequential reading from a CSV file is faster for linear conversations. Also, the quantity of dialogue is small for each simulation. Due to its simple, tabular format, the dialogues are easily editable from person to person. Each data row in the CSV file consists of the following parameter values:
- ID: Serves as a serial number for each data row
- Speaker: Denotes if the speaker is the avatar or the user. If there are multiple avatars, we write them with a serial number (for example, avatar_1, avatar_2, and so on).
- Dialogue: This refers to the dialogue spoken by the user.
- UserTurn: This is a Flag variable to check if the user is to speak. If the variable is “True”, we display a text box in which the user can speak during the simulation.
- Audio File: It contains audio of the avatar speaking when the UserTurn flag is “False”.
- Emotional Tag: This refers to the emotional tag according to which BlendShape assigns expressions to the avatar.
Figure 5 depicts a sample dataset for the interview simulation. We can see two avatars apart from the user since the interview panel consists of two people.

We initially load and parse the CSV file for each simulation scenario. When we start playing a scenario, the simulation reads each data row in the CSV file sequentially. This is because the CSV file depicts the gradual flow of a conversation between the avatar(s) and the user. We verify the UserTurn parameter in the CSV file, which is set to either True or False. If True, it indicates that it is the user’s turn to speak, and a text box appears on the screen showing what the user must say. We use the TextMeshPro UI tools to add the text box and position it at the bottom of the simulation. If False, it means it is the avatar’s turn to speak, and we play an audio file corresponding to the data row. In addition to dialogue and turn-taking information, each CSV row also contains an EmotionalTag that links the scripted line to the BlendShape-driven expression system. When the simulation encounters an EmotionalTag, the system uses it to retrieve the correct BlendShape configuration defined in the BlendShape tool. A custom controller script then activates the relevant BlendShape parameters at the moment the avatar speaks that line, producing an expression that aligns precisely with the emotional intent of the dialogue. Figure 6 depicts the composite UI, showing a fellow customer in the Grocery Store scenario with a neutral expression. The dialog box below shows the response the user must speak to converse with the avatar. The presence of a well-structured CSV file and sequential reading keeps the dialogue between the avatar(s) and the user quick and smooth. The audio files are stored in the Resources folder in Unity to be easily accessed when the CSV file is parsed.
 and [Advanced People Pack](https://assetstore.unity.com/packages/3d/characters/humanoids/humans/advanced-people-pack-2-170756) via [Unity Asset Store](https://assetstore.unity.com/) and are subject to the [Unity Asset Store](https://assetstore.unity.com/) EULA.](https://www.frontiersin.org/files/Articles/1801743/xml-images/frvir-07-1801743-g006.webp)
3.2 Gamification
Gamification refers to the use of video-game elements within a non-game system to improve user experience (Deterding et al., 2011). The primary purpose of gamification is to motivate players by directing and enhancing their focus on achieving a specific goal. In this work, gamification is incorporated through progressive scenario difficulty, task-oriented social conversations, and performance-based progression within the VR environment. Central to this design is Goal-Setting Theory, which posits that clearly defined goals enhance motivation and performance by giving users a concrete target and a measurable sense of progress (Locke and Latham, 2002). This goal-oriented nature provides the user with extrinsic motivation and is a key game element we integrate into our simulation.
Complementing this, Self-Determination Theory argues that intrinsic motivation increases when activities satisfy the psychological need of competence that is embedded within VR tasks through user choice and realistic social interactions. Alongside Flow Theory, it highlights the importance of balancing skill and challenge to achieve deep immersion and focused engagement (Csikszentmihalyi, 1990). Accordingly, the simulation scenarios were designed with progressively increasing social and cognitive demands to maintain engagement while avoiding abrupt difficulty escalation. In the following project, we create scenarios where we facilitate flow by regulating the difficulty level and offering feedback. Games that induce flow lead to improved performance, increased motivation, and heightened player satisfaction (Nacke et al., 2009).
In our system, HRV is used as a psychophysiological indicator to regulate challenge levels and help users remain within the optimal flow zone defined by Flow Theory. Stable or increasing HRV values typically reflect effective emotional regulation and manageable cognitive load, signaling that the ongoing task difficulty aligns well with the user’s current capabilities (Kim et al., 2018). Under these conditions, the system maintains or gradually increases goal difficulty to sustain engagement. In contrast, decreasing HRV suggests elevated stress or overload. HRV measurements and drops were used to evaluate whether the progressively structured scenarios elicited differentiated physiological responses consistent with their intended difficulty levels. Scenarios involving greater conversational complexity, longer interaction duration, and higher levels of social evaluation were expected to produce stronger HRV reductions relative to baseline conditions (Immanuel et al., 2023). The gamification structure of the simulation is reflected in the progressive organization of the scenarios. Each level introduces increasing conversational demands, social pressure, and task complexity. This creates a structured challenge progression similar to difficulty scaling in traditional games. This progression was intended to sustain participant engagement while eliciting varying levels of affective response throughout the simulation.
HRV reflects the variation in time intervals between consecutive heartbeats and is commonly used as an indirect indicator of autonomic nervous activity. It is influenced by the balance between parasympathetic (“rest-and-digest”) and sympathetic (“fight-or-flight”), which together regulate cardiac function. Lower HRV is often associated with increased physiological stress and reduced autonomic flexibility, whereas higher HRV is generally observed during more relaxed states. In this study, HRV is measured using the Standard Deviation of Normal-to-Normal intervals (SDNN) derived from Apple Watch sensor data. SDNN is used here as a descriptive physiological measure to capture variations in autonomic arousal during exposure to different social interaction scenarios within the simulation. HRV is not used as a measure of social skills, confidence, or performance. Instead, it serves as a physiological signal to examine whether the simulation elicits differentiated autonomic responses across scenarios with varying social demands.
We associate difficulty levels with different scenarios based on the skills tested and the conversation within the simulation. This helps us establish a flow, since a sufficient score is required to unlock the next scenario. As discussed above, each scenario comprises five scenes where a mixture of competencies is tested. For example, the Networking event scenario is the toughest since it includes testing various competencies over longer intervals. Based on the effort required, we establish the difficulty level of each scenario as follows:
- Level 1 refers to the Store visit scenario where you initiate conversations with workers or engage in conversations with a neighbourhood friend. These conversations are typically friendly and do not require speaking continuously.
- Level 2 refers to the Fancy restaurant scenario, where you initiate conversations with the waiter or engage in an intense conversation with a group of friends. The tone of the environment is more sophisticated, and a certain decorum is required while conversing with each other, making it more challenging.
- Level 3 refers to the Job Interview scenario, where you focus on dealing with authoritative figures. This is designed within a formal setting, making it more pressuring for the user. The user must read the answers to the questions, some of which are detailed, long answers. The user requires more focus for a longer period and needs to speak a lot more, making it more challenging than Levels 1 and 2.
- Level 4 refers to the Networking event scenario, where you focus on every conversation skill mentioned above. This is also designed within a formal setting and requires the user to participate in group interactions, initiate conversations with recruiters, pitch information about themselves, and deal with company rejections. This scenario is considered the most difficult since it has the broadest number of use cases, making it difficult for users to maintain their composure.
4 Methods section
This section describes the participant recruitment, experimental design, and data collection procedures used to evaluate physiological responses during VR-based social interaction scenarios. The study focuses on measuring HRV as an indicator of autonomic response under controlled simulated conditions. Details regarding participants, experimental protocol, VR environment, and data handling are outlined below to ensure reproducibility and transparency.
4.1 Participants
A total of 19 participants were recruited for this study, comprising current students and alumni from Stevens Institute of Technology. Participants were between 25 and 35 years of age and included individuals of mixed gender. All participants had a background in computer science or related engineering disciplines and expressed an interest in virtual reality (VR) and machine learning research. Inclusion criteria required participants to have prior experience in technology-related fields. Exclusion criteria included individuals without a background in computer science or related engineering domains. The selected sample was considered sufficient to provide initial insights into construct validity. The participants were recruited through word-of-mouth, email communication, and text-based outreach, leveraging existing academic and professional networks within Hudson County. Identifiable data and informed consent were collected via Google Forms before participation. Participation was voluntary.
4.2 Physiological measures
Heart Rate Variability (HRV) was used as the primary physiological measure. A custom watchOS application developed in Swift was used to continuously record raw RR intervals from the smartwatch sensor. The application computes the Standard Deviation of NN intervals (SDNN) as the primary HRV metric. Baseline and experimental HRV were recorded using the same device to ensure consistency. HRV data was sampled at the intervals using the Equation 1 denoted below:where x represents scenario duration (minutes), y represents additional seconds, and z represents the number of HRV samples. Each scenario was summarized using z = 12 samples, resulting in a sampling interval of approximately 15–20 s for 3–4 min scenarios. This sampling rate balances temporal resolution with noise reduction. HRV changes were quantified relative to baseline, with task-related HRV expected to decrease under stress. A stress threshold was defined as 0.70 × baseline HRV, representing moderate stress while maintaining individualized scaling across participants. A baseline-relative threshold was adopted because resting HRV varies considerably across individuals, making absolute thresholds less suitable for participant comparisons. The 70% threshold was selected as a conservative criterion that exceeds minor physiological fluctuations while remaining sensitive to meaningful stress-related changes. Consequently, the threshold provides a practical balance between over-classifying normal variability and under-identifying genuine physiological responses.
Δ t = x × 60 + y z (1)
4.3 Study design and procedure
The study employed a repeated-measures experimental design in which participants completed multiple simulated social interaction scenarios while physiological data were continuously recorded. This is especially useful in measuring change over the various simulation scenarios for each participant. Each session consists of:
- A 5-min resting period to measure initial (baseline) HRV value
- Each scenario lasts for 3–4 min, where the change in HRV is measured
- Participants were provided rest periods between scenarios. The duration of the rest period was determined by the participant’s physiological response rather than by the scenario itself. Specifically, when HRV measurements met the predefined stress criterion, a 5-min recovery period was provided. Otherwise, a 1-min transition period was used. These intervals were intended to reduce participant fatigue and provide an opportunity for recovery before proceeding to the next scene. The recovery protocols were implemented to reduce the likelihood of carryover effects, although physiological recovery to baseline was not formally verified.
To reduce confounding physiological effects, participants were instructed to avoid nicotine, caffeine, and physical exercise for at least 1 hour before the session. Additionally, participants completed a pre-simulation questionnaire capturing information about their prior experience with VR and their routine before arriving for the experiment. For analytical consistency, each scenario was evaluated relative to the participant’s baseline HRV, which served as an individualized reference.
4.4 Data handling and privacy
We have received IRB approval for the collection and preservation of the HRV data. Identifiable data collected during recruitment is stored separately from physiological data and destroyed after study completion. All HRV data is de-identified prior to storage. Data is securely stored in a password-protected cloud environment with restricted access and multi-factor authentication. The experiments were conducted in a controlled lab setting, with participants tested individually to ensure privacy. De-identified data may be shared for research purposes without the possibility of re-identification.
4.5 Participant safety
Participant safety was prioritized throughout the study. Individuals completed a screening procedure and received an orientation session describing the VR environment, study protocol, available controls, and withdrawal procedures. Participants were informed about the nature of the VR-based social interaction tasks and their right to pause or withdraw from the study at any time without penalty. They could do so by verbally notifying the researcher or removing the VR headset directly. As discussed earlier, the protocol consisted of four short VR scenarios lasting approximately 3–4 min each, with rest periods provided between scenarios to minimize discomfort. Throughout the experiment, participants were continuously monitored by the research team. They were encouraged to immediately report any signs of distress, and periodic verbal checks were conducted during the session. The experiment would be paused or terminated upon participant request or if the researcher observed physical signs like sweating, shivering, breathlessness, and so on.
Participants did not report significant physical discomfort or VR-related adverse effects throughout the study. Informal feedback after the VR simulation suggested that participants generally perceived the scenarios as socially and cognitively demanding rather than physically uncomfortable. Following completion of the study, participants could discuss their experience and ask further questions about their data. Their wellbeing was confirmed before leaving the laboratory.
5 Results
Based on the experimental design ideologies discussed in the previous section, we developed a custom watchOS application for real-time HRV logging and recording HRV changes. The application was implemented in Swift and continuously records raw RR intervals from the Apple Watch heart sensor during experimental scenarios. HRV during task engagement was quantified using SDNN, a standard time-domain metric computed from successive RR intervals. Baseline HRV was obtained directly from the Apple Watch during a designated resting period and used as the reference HRV for each participant. Since Apple Watch reports HRV using the SDNN metric, the baseline value is directly comparable to the SDNN values computed from RR intervals during the experimental scenarios. Thus, all HRV measurements used throughout the study are expressed using the same underlying HRV metric. The SDNN estimates computed by the custom application represent the scenario values, and the smartwatch-provided SDNN estimate represents the baseline value. RR intervals derived from the Apple Watch was consistently used within the custom application for all SDNN computations to ensure methodological consistency. The HRV drops during the experimental scenarios were then computed relative to this baseline HRV.
Using the baseline provided by the smartwatch allows for a stable, participant-specific reference while enabling consistent quantification of HRV changes across scenarios. We record changes in the HRV based on the calculated sampling interval of 15–20 s and append these values to a.csv file for each scenario. Once we receive the 12 entries throughout the scenario, we calculate the average, which gives us an aggregate estimate of how each participant handles various social situations in each scenario. The scenario-wise HRV value drops, and the averages for a particular participant are summarized in Table 1.
TABLE 1. HRV measurements across baseline and simulation scenarios for a sample participant.
| Reading | Baseline | Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4 |
|---|---|---|---|---|---|
| 1 | 70 | 69 | 68 | 69 | 68 |
| 2 | 70 | 67 | 66 | 66 | 67 |
| 3 | 70 | 66 | 65 | 65 | 65 |
| 4 | 70 | 66 | 64 | 66 | 63 |
| 5 | 70 | 63 | 61 | 61 | 61 |
| 6 | 70 | 63 | 61 | 60 | 60 |
| 7 | 70 | 59 | 57 | 58 | 59 |
| 8 | 70 | 57 | 56 | 56 | 55 |
| 9 | 70 | 55 | 53 | 54 | 52 |
| 10 | 70 | 54 | 52 | 51 | 51 |
| 11 | 70 | 52 | 50 | 48 | 43 |
| 12 | 70 | 48 | 48 | 45 | 41 |
| Avg | 70 | 59.92 | 58.42 | 58.08 | 57.08 |
Table 1 presents an example of HRV measurements for a representative participant across the baseline condition and the four simulated social scenarios. Scenario one corresponds to the Store Visit, Scenario two to the Fancy Restaurant, Scenario three to the Job Interview, and Scenario four to the Networking Event. Baseline HRV remains stable across readings, reflecting a resting reference condition. In contrast, HRV values recorded during the scenarios show a progressive reduction relative to baseline, with greater decreases observed in scenarios associated with higher social and cognitive demand. The averaged HRV values highlight this trend, with lower mean HRV observed for the Job Interview and Networking Event scenarios compared to the Store Visit and Fancy Restaurant scenarios. This table illustrates how HRV changes were quantified relative to baseline across multiple time samples within each scenario.
The evaluation of the simulation was conducted using two complementary criteria: affective fidelity and construct validity. Affective fidelity refers to the extent to which the simulation elicits emotional responses comparable to those experienced in real-world social situations and was assessed using heart rate variability (HRV) as a physiological indicator of affective response. Affective fidelity was primarily evaluated using aggregate HRV responses from all 19 participants. Figure 7 shows the net drop obtained by subtracting the baseline average from the average HRV for each scenario. The observed reductions for the participants demonstrate that the simulation elicits measurable physiological responses for varying scenarios. This data is crucial for demonstrating that the simulation affects all participants to varying degrees, thereby demonstrating affective fidelity.

To further illustrate how HRV changed throughout the simulation at the individual level, we additionally present a detailed single-participant HRV analysis. Participant 13 was selected because they exhibited measurable HRV reductions across all four scenarios, thereby providing a consistent example of physiological responsiveness throughout the experimental session. This participant was not intended to represent the average participant response. Rather, the visualization illustrates scenario-dependent HRV variation over time. Figure 8 illustrates the normalized HRV trajectory for this participant across the four scenarios, showing a pronounced reduction in HRV relative to the baseline during simulated interactions. A gradient trend line is included to connect scenario-wise HRV reductions back to the baseline HRV, reflecting our experimental assumption that the rest intervals between scenarios allow autonomic activity to recover toward baseline levels. Among the evaluated scenarios, the Networking Event (Scenario 4) exhibits the largest decline from baseline HRV. Furthermore, the overall downward trend across scenarios indicates progressively increasing social and cognitive demands, with later scenarios eliciting stronger affective responses than earlier ones. Differences in the slopes of HRV reduction across scenarios further highlight variations in affective engagement elicited by each simulated context.

Construct validity refers to the degree to which a simulation accurately represents the intended task demands and is sensitive to meaningful differences in participant responses. In the context of this study, the intended construct is the elicitation of physiological stress responses during social interaction. In contrast to affective fidelity, which focuses on within-participant affective trends, construct validity requires examining inter-individual variability across both participants and scenarios. Quantitatively, this variability was evaluated using the mean and standard deviation of participant HRV responses across the four scenarios. The raw mean HRV values provide an overall representation of the physiological state observed during each scenario. The corresponding standard deviations indicate the degree of variability in participant responses around these averages. By considering the mean HRV values together with their associated standard deviations, we obtain a quantitative representation of the distribution and range of physiological responses across participants. Figure 9 presents the quantitative representation of construct validity through the range of HRV responses observed among the 19 participants. For the 3-min scenarios evaluated in this study, the observed variability and aggregate HRV trends were considered satisfactory across all four scenarios. These findings suggest that the simulation is capable of eliciting measurable physiological stress responses while remaining sensitive to individual differences in participant reactivity.

In addition to the raw HRV measurements, baseline-relative HRV drops were analyzed to quantify the magnitude of physiological response induced by each scenario. The mean HRV drops represent the average reduction in HRV relative to participant baseline levels and therefore serve as an indicator of affective engagement. The progressively decreasing mean HRV across scenarios suggests that scenarios with higher cognitive demands elicit stronger physiological responses, thereby supporting affective fidelity. Furthermore, the standard deviations associated with the HRV drops indicate variability in individual responsiveness for participants. This supports construct validity by demonstrating that the simulation captures meaningful participant-specific differences rather than producing uniform reactions across all users. Table 2 depicts the mean, standard deviation, and 95% Confidence Intervals (CI) of the HRV drops to strengthen the interpretation. The relatively narrow confidence intervals across conditions indicate stable estimation of mean HRV responses. The progressive shift in interval ranges across scenarios further reinforces the graded increase in physiological stress. Therefore, the table collectively indicates both affective fidelity and construct validity.
TABLE 2. HRV drop statistics across simulation scenarios.
| Scenario | Mean HRV drop | Std. Deviation | 95% CI |
|---|---|---|---|
| Store visit | 4.70 | 2.37 | [3.56, 5.84] |
| Fancy restaurant | 6.26 | 3.09 | [4.77, 7.75] |
| Job interview | 9.73 | 4.33 | [7.64, 11.81] |
| Networking event | 13.98 | 4.07 | [12.02, 15.94] |
HRV responses from 19 participants were analyzed across all scenarios to derive the qualitative representation of the simulation. Rather than assuming uniform reactivity, a conservative HRV cutoff was applied to determine whether a participant exhibited a measurable physiological response during a given scenario. Specifically, participants whose HRV dropped below the predefined threshold relative to resting levels were classified as being affected by that scenario. Figure 10 illustrates the resulting distribution of participant responses, revealing substantial variability. Some participants were unaffected across all scenarios, while others showed sensitivity to one, multiple, or all scenarios. The Networking Event affected the largest proportion of participants, whereas the Store Visit affected the fewest, aligning with the relative social and cognitive demands of each task. The Friedman test yielded a statistic of 34.43 with three degrees of freedom and a p-value of less than 0.001. This indicates a statistically significant difference in HRV responses across the four scenarios. The corresponding effect size, measured using Kendall’s coefficient of concordance (W = 0.61), indicates a large effect in our construct, which is eliciting physiological responses observed across participants. To further identify which scenarios differed from one another, post hoc Wilcoxon signed-rank tests were conducted following the Friedman test. The test revealed significant pairwise differences between multiple scenario combinations, and the results of the test are depicted in Table 3. This table complements the descriptive findings in Figure 10 by confirming that the differences in participant responsiveness across scenarios are not only visually observable but also statistically significant at the group level. Importantly, the affected variability shown in Figure 10 is based on a heuristic baseline threshold and is intended solely as a descriptive visualization of participant-level physiological variation rather than a basis for statistical inference.

TABLE 3. Post-hoc wilcoxon signed-rank test results with bonferroni correction.
| Comparison | W statistic | p-value | Bonferroni p-value | Significant |
|---|---|---|---|---|
| Store visit vs. fancy restaurant | 64.5 | 0.1305 | 0.7830 | No |
| Store visit vs. job interview | 11.0 | 0.0004 | 0.0027 | Yes |
| Store visit vs. networking event | 7.0 | 0.0000 | 0.0001 | Yes |
| Fancy restaurant vs. job interview | 25.5 | 0.0018 | 0.0105 | Yes |
| Fancy restaurant vs. networking event | 3.0 | 0.0001 | 0.0006 | Yes |
| Job interview vs. networking event | 17.0 | 0.0006 | 0.0037 | Yes |
6 Discussion
The results demonstrate that immersive social simulations can reliably elicit measurable physiological stress responses captured through HRV. Across participants, a consistent reduction in HRV was observed during task engagement relative to baseline. The largest declines occurred in the Job Interview and Networking Event scenarios. These findings are consistent with established social evaluative literature, which shows that HRV decreases under conditions of performance pressure. The stronger HRV reductions observed can also be interpreted through the lens of social evaluative threat theory. The theory posits that formal situations involving judgment significantly heighten physiological stress responses. These scenarios explicitly required participants to engage in structured verbal performance under perceived evaluation. In contrast, the Store Visit and Fancy Restaurant scenarios produced comparatively milder HRV reductions, suggesting lower levels of social threat and reduced performance pressure. This gradient of physiological response aligns with prior findings that social complexity and perceived judgment intensity are key determinants of autonomic stress reactivity. This observed pattern is further supported by the Friedman test, which reveals significant differences in HRV responses across scenarios (p less than 0.05). This indicates that not all conditions elicit equivalent physiological responses. Post-hoc Wilcoxon tests further demonstrate that the Job Interview and Networking Event scenarios were significantly different from the lower-stress scenarios. On the other hand, the Store Visit and Fancy Restaurant did not significantly differ from each other, suggesting a clear low-stress versus high-stress clustering effect.
From a design perspective, these effects can be directly linked to the structure of the simulation environment. The progression from informal to formal social scenarios introduced a gamified escalation in difficulty, with each successive task increasing in social complexity and cognitive demand. Specifically, the simulation incorporated structured conversational objectives, progressively scaled challenges, and increasing social task demand that collectively reflect core gamification principles. The use of gamified progression, while enhancing engagement, may also have intensified performance results. The progressively increasing mean HRV drops across scenarios suggest that these gamified interaction structures contributed to heightened affective engagement beyond passive VR exposure alone. The recovery patterns between scenarios further indicate that the design allowed partial autonomic resetting, although not always to baseline levels, suggesting residual arousal effects. This supports the notion that sequential social challenges may produce cumulative stress effects, even when separated by rest intervals. In this study, the observed HRV sensitivity also suggests that the VR simulation elicits stress responses resembling those observed in real-world settings. This is particularly important given that traditional survey-based or non-immersive assessments often fail to capture such dynamic autonomic fluctuations. Additionally, the inclusion of 95% confidence intervals around the mean HRV drops provides further evidence of the stability and precision of these observed effects. This reinforces that the graded physiological response across scenarios is not only consistent but also reliably estimated across participants.
The following study also provides the necessary results, showcasing the responsiveness for a single participant and comparing multiple participants’ Net HRV drop. This shows that the participants are indeed affected by the simulation. On the other hand, inter-participant variability further supports the construct validity of the simulation. Apart from that, the mean and the standard deviation of the HRV drops indicate both affective fidelity and construct validity. Not all participants exhibited uniform HRV reductions across scenarios, reflecting individual differences in stress sensitivity, social comfort, and cognitive appraisal of evaluative situations. This variability is consistent with previous research showing that physiological responses to social interaction are highly individualized and may vary considerably across participants. Overall, these findings suggest that the simulation successfully captures both within-subject emotional dynamics and between-subject variability, thereby demonstrating strong affective fidelity and construct validity. Moreover, the integration of gamified progression and immersive VR elements appears to meaningfully shape physiological responses in a manner consistent with established theories of performance-based stress. Consequently, support for construct validity is derived primarily from the Friedman and Wilcoxon analyses, which demonstrate differentiated physiological responses across social interaction scenarios. It demonstrates that participant responses are not random but systematically structured across social contexts, thereby providing evidence for construct validity.
7 Limitations and future work
Despite designing and evaluating the VR simulation, we would like to acknowledge the following limitations. Firstly, the simulation is most effective for participants during their debut, since using it repeatedly will give them a mental picture of how the conversations progress. The conversations are pre-scripted, making the simulation repetitive, affecting the participant’s affective fidelity. This suggests that the simulation’s effectiveness decreases with the number of times participants use it. As part of our future work, we plan to resolve the issue using Natural Language Processing (NLP) models that provide an end-to-end dialogue between the avatars and the users. This would include using Speech-to-Text (STT) and Text-to-Speech (TTS) alongside dialogue management models for real-time text generation. We can use the GPT API, and Google STT and TTS to create a chatbot system between the user and the avatar (Pelrine et al., 2023). This extension would potentially improve realism and interaction variability, thereby enhancing the ecological validity of the simulation.
The second limitation arises from the reliance on HRV as a measure of the individual’s confidence within the simulation. However, HRV alone does not provide a direct or subjective measure of anxiety. It should rather be interpreted as an indicator of physiological stress responses. Factors such as sleep, hygiene, and caffeine consumption can easily influence HRV. Although we take measures to limit these influences by providing guidelines, we cannot constantly monitor the participants. We mention the required instructions in the pre-experiment questionnaire and request a verbal confirmation of the same. We also rely on two varying sources to acquire the HRV data which is the custom application and Apple Watch. The baseline HRV relies on the Apple Watch’s proprietary SDNN implementation, whereas scenario HRV was computed using a custom SDNN pipeline. Even though the RR intervals are consistent for both, minor differences in preprocessing and filtering cannot be fully excluded. Furthermore, structured rest intervals were incorporated between scenarios to facilitate physiological recovery and minimize carryover effects. The return to baseline HRV before each subsequent scenario was not formally verified. Consequently, residual physiological arousal between scenarios may have influenced the observed HRV responses and should be considered when interpreting the results. Future studies should incorporate objective verification of physiological recovery prior to each scenario to better isolate the effects of individual social interactions. Other measurements of emotional impact include Galvanic Skin Response (GSR), Electroencephalography (EEG), Eye Tracking, Facial Expression Analysis, and Voice Stress Analysis. Rather than overly relying on HRV data, we plan to research physiological parameters, such as facial expressions (Pediaditis et al., 2015) and voice (Sondhi et al., 2015), to enhance the existing study. We shall use computer vision and voice analytics to conduct and integrate this research with the simulation.
Finally, this study requires a larger sample size of participants. The current study evaluates 19 participants, which needs to be expanded to a larger population for more concrete results. We also require more assessment of individuals, their learning rates, and the consistency of their responses with the simulation. Although the current pre-scripted dialogue includes 3 min of runtime per scene, we can consider that the duration of the intervention can also be increased. This is because a greater length of time can allow further fluctuation in HRV and rigorous training of participants. The use of pre-scripted dialogue also affects the long-term efficacy since we use the same script repeatedly. Although each scenario comprises multiple scenes, a certain time period will cause the scenes to repeat for a given participant. This is why the addition of dynamic conversations is also necessary to integrate within the simulation. The proposed system can be adapted for broader populations by leveraging its dynamic dialogue generation and physiological feedback mechanisms to personalize interactions based on individual user profiles. However, this remains a potential future application and has not been validated in the current study. While making these significant additions, we shall ensure that our experimental rigor is consistent with the current methodology for consistency and valid comparisons.
8 Conclusion
This study presented a VR-based social simulation system designed to measure real-time affective responses using a physiological indicator like HRV. The results demonstrate consistent reductions in HRV during socially demanding scenarios. This is observed particularly in high-evaluation contexts such as job interviews and networking events. These findings indicate that the system elicits measurable physiological responses, with more demanding scenarios generally associated with greater reductions in HRV. The observed HRV dynamics suggest affective fidelity, as participants exhibited measurable autonomic changes consistent with real-world social stressors. Across both individual and group analyses, the simulation demonstrated sensitivity to scenario difficulty and participant variability. This supports the construct validity of the simulation in eliciting measurable physiological stress responses. The statistical analyses further confirmed significant differences in physiological responses across scenarios. The post hoc comparisons reveal clear distinctions between the lower-stress and higher-stress social interactions. Furthermore, the progressively increasing HRV reductions observed across scenarios suggest that the structured gamified progression may contribute to heightened affective engagement within immersive social environments. The inclusion of confidence interval analysis also provided additional evidence regarding the consistency and reliability of the observed HRV responses across participants. Overall, the integration of immersive VR and structured gamified progression provides a promising framework for studying social stress in controlled and meaningful environments.
Generative AI statement
The author(s) declared that generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
Edited by: Artur Pilacinski, Ruhr University Bochum, Germany
Reviewed by: Timothy John Pattiasina, Institut Informatika Indonesia, Indonesia
Silvino Martins, University of Aveiro, Portugal
Luís Ferreira, Instituto Politecnico do Cavado e do Ave Laboratorio de Inteligencia Artificial Aplicada, Portugal
Statements
- Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
- Funding
The author(s) declared that financial support was not received for this work and/or its publication.
- Data availability
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
- Ethics approval
The studies involving humans were approved by Stevens Institute of Technology IRB. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
- Author contributions
PP: Writing – original draft, Writing – review and editing, Investigation, Supervision, Formal Analysis, Software, Validation, Data curation, Methodology, Conceptualization, Visualization. OO: Project administration, Supervision, Funding acquisition, Writing – review and editing, Writing – original draft, Resources.
References
- Anderson P. L., Price M., Edwards S. M., Obasaju M. A., Schmertz S. K., Zimand E. (2013). Virtual reality exposure therapy for social anxiety disorder: a randomized controlled trial. J. Consulting Clinical Psychology 81 (5), 751 – 760. doi:10.1037/a0033559
- Andersson J., Hallin J., Tingström A., Knutsson J. (2024). Virtual reality exposure therapy for fear of spiders: an open trial and feasibility study of a new treatment for arachnophobia. Nordic J. Psychiatry 78 (2), 128 – 136. doi:10.1080/08039488.2023.2279643
- Barlow D. H. (2004). Anxiety and its Disorders: The Nature and Treatment of Anxiety and Panic. New York, NY: Guilford Press.
- Beck J. S. (2011). Cognitive-behavioral therapy. Clin. Textbook Addictive Disorders 491, 474 – 501.
- Beidel D. C., Alfano C. A., Kofler M. J., Rao P. A., Scharfstein L., Sarver N. W. (2014). The impact of social skills training for social anxiety disorder: a randomized controlled trial. J. Anxiety Disorders 28 (8), 908 – 918. doi:10.1016/j.janxdis.2014.09.016
- Bouchard S., Dumoulin S., Robillard G., Guitard T., Klinger E., Forget H. (2011). A randomized control trial for the use of in virtuo exposure in the treatment of social phobia: final results. J. CyberTherapy Rehabilitation 4 (2), 197 – 200.
- Bouchard S., Dumoulin S., Robillard G., Guitard T., Klinger E., Forget H. (2017). Virtual reality compared with in vivo exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. Br. J. Psychiatry 210 (4), 276 – 283. doi:10.1192/bjp.bp.116.184234
- Bright E., Vine S., Wilson M. R., Masters R. S., McGrath J. S. (2012). Face validity, construct validity and training benefits of a virtual reality TURP simulator. Int. J. Surg. 10 (3), 163 – 166. doi:10.1016/j.ijsu.2012.02.012
- Caponnetto P., Triscari S., Maglia M., Quattropani M. C. (2021). The simulation game—virtual reality therapy for the treatment of social anxiety disorder: a systematic review. Int. Journal Environmental Research Public Health 18 (24), 13209. doi:10.3390/ijerph182413209
- careerpotential (2026). Wordfence is a security plugin installed on over 5 million WordPress sites. The owner of this site is using Wordfence to manage access to their site. Available online at: (Accessed August 17, 2026).
- Csikszentmihalyi M. (1990). Flow: The Psychology of Optimal Experience.
- Deterding S., Sicart M., Nacke L., O’hara K., Dixon D. (2011). “ Gamification. using game-design elements in non-gaming contexts,” in CHI’11 Extended Abstracts on Human Factors in Computing Systems, 2425 – 2428.
- Digital Innovation and Transformation (2008). Psious – using VR exposure therapy to overcome fears & anxiety disorders. Available online at: (Accessed August 17, 2026).
- Ferreira L., Cerqueira J., Jonassi J., Jesus A., Amaral C., Mateus-Coelho N. (2024). Virtual reality on public speaking phobia mitigation. Procedia Comput. Sci. 239, 2251 – 2259. doi:10.1016/j.procs.2024.06.416
- Gorini A., Riva G. (2008). Virtual reality in anxiety disorders: the past and the future. Expert Rev. Neurother. 8 (2), 215 – 233. doi:10.1586/14737175.8.2.215
- Grenier S., Forget H., Bouchard S., Isere S., Belleville S., Potvin O. (2015). Using virtual reality to improve the efficacy of cognitive-behavioral therapy (CBT) in the treatment of late-life anxiety: preliminary recommendations for future research. Int. Psychogeriatrics 27 (7), 1217 – 1225. doi:10.1017/S1041610214002300
- Harris D. J., Bird J. M., Smart P. A., Wilson M. R., Vine S. J. (2020). A framework for the testing and validation of simulated environments in experimentation and training. Front. Psychol. 11, 605. doi:10.3389/fpsyg.2020.00605
- Immanuel S., Teferra M. N., Baumert M., Bidargaddi N. (2023). Heart rate variability for evaluating psychological stress changes in healthy adults: a scoping review. Neuropsychobiology 82 (4), 187 – 202. doi:10.1159/000530376
- James A. C., Reardon T., Soler A., James G., Creswell C. (2020). Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database Systematic Reviews 11 (11), CD013162. doi:10.1002/14651858.CD013162.pub2
- Kahlon S. (2022). Virtual Reality Exposure Therapy for Adolescents with Public Speaking Anxiety.
- Kampmann I. L., Emmelkamp P. M., Hartanto D., Brinkman W. P., Zijlstra B. J., Morina N. (2016). Exposure to virtual social interactions in the treatment of social anxiety disorder: a randomized controlled trial. Behav. Research Therapy 77, 147 – 156. doi:10.1016/j.brat.2015.12.016
- Ke F., Moon J., Sokolikj Z. (2022). Virtual reality–based social skills training for children with autism spectrum disorder. J. Special Educ. Technol. 37 (1), 49 – 62. doi:10.1177/0162643420945603
- Kim H. G., Cheon E. J., Bai D. S., Lee Y. H., Koo B. H. (2018). Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry Investig. 15 (3), 235 – 245. doi:10.30773/pi.2017.08.17
- Krijn M., Emmelkamp P. M., Olafsson R. P., Biemond R. (2004). Virtual reality exposure therapy of anxiety disorders: a review. Clin. Psychology Review 24 (3), 259 – 281. doi:10.1016/j.cpr.2004.04.001
- Li C., Liang W., Quigley C., Zhao Y., Yu L.-F. (2017). Earthquake safety training through virtual drills. IEEE Trans. Vis. Comput. Graph. 23, 1275 – 1284. doi:10.1109/tvcg.2017.2656958
- Li X., Yi W., Chi H. L., Wang X., Chan A. P. (2018). A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Automation Construction 86, 150 – 162. doi:10.1016/j.autcon.2017.11.003
- Lin F., Ye L., Duffy V. G., Su C. J. (2002). Developing virtual environments for industrial training. Inf. Sciences 140 (1-2), 153 – 170.
- Lindner P., Miloff A., Fagernäs S., Andersen J., Sigeman M., Andersson G. (2019). Therapist-led and self-led one-session virtual reality exposure therapy for public speaking anxiety with consumer hardware and software: a randomized controlled trial. J. Anxiety Disorders 61, 45 – 54. doi:10.1016/j.janxdis.2018.07.003
- Liu D., Yu J., Macchiarella N. D., Vincenzi D. A. (2008). “ Simulation fidelity,” in Human Factors in Simulation and Training (Boca Raton, FL: CRC Press), 91 – 108.
- Locke E. A., Latham G. P. (2002). Building a practically useful theory of goal setting and task motivation: a 35-year odyssey. Am. Psychologist 57 (9), 705 – 717. doi:10.1037//0003-066x.57.9.705
- López-López J. A., Davies S. R., Caldwell D. M., Churchill R., Peters T. J., Tallon D. (2019). The process and delivery of CBT for depression in adults: a systematic review and network meta-analysis. Psychol. Med. 49 (12), 1937 – 1947. doi:10.1017/S003329171900120X
- Nacke L., Drachen A., Kuikkaniemi K., Niesenhaus J., Korhonen H. J., Hoogen W. M. (2009). “ Playability and player experience research,” in Proceedings of Digra 2009: Breaking New Ground: Innovation in Games, Play, Practice and Theory.
- Nor N., Sunar M., Kapi A. (2019). A review of gamification in virtual reality (VR) sport. EAI Endorsed Trans. Creative Technol. 6 (21), 163212. doi:10.4108/eai.13-7-2018.163212
- Olivares-Olivares P. J., Ortiz-González P. F., Olivares J. (2019). Role of social skills training in adolescents with social anxiety disorder. Int. Journal Clinical Health Psychology 19 (1), 41 – 48. doi:10.1016/j.ijchp.2018.11.002
- Park K. M., Ku J., Choi S. H., Jang H. J., Park J. Y., Kim S. I. (2011). A virtual reality application in role-plays of social skills training for schizophrenia: a randomized, controlled trial. Psychiatry Research 189 (2), 166 – 172. doi:10.1016/j.psychres.2011.04.003
- Pattiasina T. J., Rosyid H. A., Handayani A. N., Junaedi H., Trianto E. M., Sutjiadi R. (2025). Towards digital therapeutics: developing an immersive serious game for social anxiety disorder. Telfor J. 17 (2), 22 – 27. doi:10.5937/telfor2502022p
- Pediaditis M., Giannakakis G., Chiarugi F., Manousos D., Pampouchidou A., Christinaki E. (2015). “ Extraction of facial features as indicators of stress and anxiety,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE), 3711 – 3714.
- Pelrine K., Taufeeque M., Zajac M., McLean E., Gleave A. (2023). Exploiting Novel gpt-4 Apis.
- Rizzo A., John B., Newman B., Williams J., Hartholt A., Lethin C. (2013). Virtual reality as a tool for delivering PTSD exposure therapy and stress resilience training. Mil. Behav. Health 1 (1), 52 – 58. doi:10.1080/21635781.2012.721064
- Rothbaum B. O., Hodges L. F., Kooper R., Opdyke D., Williford J. S., North M. (1995). Virtual reality graded exposure in the treatment of acrophobia: a case report. Behav. Therapy 26 (3), 547 – 554. doi:10.1016/s0005-7894(05)80100-5
- Roy-Byrne P. P., Craske M. G., Stein M. B. (2006). Panic disorder. Lancet 368 (9540), 1023 – 1032. doi:10.1016/S0140-6736(06)69418-X
- Slater M., Sanchez-Vives M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robotics AI 3, 74. doi:10.3389/frobt.2016.00074
- Sondhi S., Khan M., Vijay R., Salhan A. K. (2015). Vocal indicators of emotional stress. Int. J. Comput. Appl. 122 (15), 38 – 43. doi:10.5120/21780-5056
- southwestern.edu (2025). Center for Career & Professional Development. Available online at: (Accessed August 17, 2026).
- Stein M. B., Sareen J. (2015). Generalized anxiety disorder. N. Engl. J. Med. 373 (21), 2059 – 2068. doi:10.1056/NEJMcp1502514
- Stein M. B., Stein D. J. (2008). Social anxiety disorder. Lancet 371 (9618), 1115 – 1125. doi:10.1016/S0140-6736(08)60488-2
- studocu (2022). Grocery conversations: everyday dialogues at the Store. Available online at:.
- Suyanto E. M., Angkasa D., Turaga H., Sutoyo R. (2017). Overcome acrophobia with the help of virtual reality and kinect technology. Procedia Computer Science 116, 476 – 483. doi:10.1016/j.procs.2017.10.062
- Thayer J. F., Lane R. D. (2000). A model of neurovisceral integration in emotion regulation and dysregulation. J. Affective Disorders 61 (3), 201 – 216. doi:10.1016/s0165-0327(00)00338-4
- thoughtco (2024). Conversations in restaurants for ESL beginners. Available online at: (Accessed August 17, 2026).
- Wang P., Wu P., Wang J., Chi H.-L., Wang X. (2018). A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public. Health. 15, 1204. doi:10.3390/ijerph15061204
- Xie B., Liu H., Alghofaili R., Zhang Y., Jiang Y., Lobo F. D. (2021). A review on virtual reality skill training applications. Front. Virtual Real. 2, 645153. doi:10.3389/frvir.2021.645153
- Yang J., Che X., Qu C., Di X., Liu H. (2024). Enhancing virtual reality exposure therapy: optimizing treatment outcomes for agoraphobia through advanced simulation and comparative analysis. Comput. Animat. Virtual Worlds 35 (4), e2291. doi:10.1002/cav.2291
Citation
Parekh, P. & Oyeleke, O. R. (2026). The design, gamification, and physiological evaluation of a VR social interaction simulation. Frontiers in Virtual Reality, Volume 7. https://doi.org/
- Received
- Accepted
- Published
- Volume
- 7 - 2026
Copyright
Copyright © 2026 Parekh and Oyeleke.
License: https://creativecommons.org/licenses/by/4.0/
*Correspondence: Pranav Parekh
Article metrics
- Views
- 0
- Downloads
- 0
- Citations
- 0
- Shares
- 0
