Virtual bodies, real touch: haptic social communication across VR, MR, and cross-reality

1 Introduction

Social touch is one of the most direct ways people communicate affect, attention, comfort, and social connection (Suvilehto et al., 2023; Field, 2010). A touch can signal emotions that are difficult to communicate through speech alone, and interpersonal touch plays an important role in how people regulate closeness, trust, and social meaning (Hertenstein et al., 2009; Cascio et al., 2019; Suvilehto et al., 2015). Emotions such as love, gratitude, sympathy, and anger can be decoded from touch alone (Hertenstein et al., 2006), and affective touch is supported by neurophysiological mechanisms specialized for the social and emotional qualities of skin-to-skin contact (McGlone et al., 2014). The COVID-19 pandemic and the resulting need for social distancing (Yang, 2021; Prattichizzo, 2021; De Figueiredo et al., 2021; Usher et al., 2020) significantly accelerated the adoption of virtual communication mediums and applications (Wang, 2020; Bönsch et al., 2018; Li et al., 2021; Qiu et al., 2023), which have demonstrated the potential to improve convenience and productivity in human interaction. However, most remote communication systems still rely mainly on audio, video, text, and visual embodiment. The tactile channel is usually absent or reduced to simple alerts and device vibrations (Hadi and Valenzuela, 2020; Van Erp and Toet, 2015). As a result, remote interactions often fail to support the interpersonal functions that physical touch naturally provides, including emotional expression, social bonding, and affect regulation. This can lead to touch starvation, which has been linked to reduced wellbeing, weaker bonding, and a loss of one of the most direct channels through which people regulate social and emotional connection (Gallace and Spence, 2010; Dunbar, 2010; Jakubiak and Feeney, 2017). Mediated social touch research has shown that dedicated haptic devices can transmit touch-like cues over distance and support emotional communication, connectedness, and social presence (Haans et al., 2006; Huisman, 2017; van Hattum et al., 2022). Yet much of this work still treats touch as a signal delivered through a device rather than as part of a shared embodied interaction. What is missing is not only tactile feedback, but the social context that makes touch meaningful: another person’s body, gesture, timing, intention, and response.

Unlike screen-based media, Extended Reality (XR) can place remote people into shared spatial environments where bodies, avatars, objects, gaze, and movement are coordinated in real time. In that sense, XR should not simply become a more immersive version of video calling; it should expand the social bandwidth of remote interaction. If users can see another person’s avatar, reach toward them, move around them, and interact in the same space, then touch should not remain outside the loop. Haptics has therefore become a central challenge for virtual, augmented, mixed, and extended reality systems (Slater and Sanchez-Vives, 2016; Pacchierotti et al., 2024), especially as social XR moves beyond visual co-presence toward richer forms of embodied communication.

Within the scope of XR, Virtual Reality (VR) replaces the user’s physical surroundings with a fully synthetic shared world, which can make avatar-mediated interaction visually coherent and easier to control. In contrast, Mixed Reality (MR) keeps the user anchored in their real environment while embedding virtual people and objects into that space. These differences become even more important in cross-reality interaction, where one person may be immersed in VR while the other remains situated in MR. Studying such configurations is important because social XR is unlikely to be used under perfectly matched conditions: collaborators may differ in their available hardware, physical surroundings, mobility, comfort, or ability to use a fully immersive display (Auda et al., 2023; Merz et al., 2024; Stepanova et al., 2026). Cross-reality interaction can therefore broaden participation while combining complementary affordances, allowing a VR user to enter a controlled and immersive shared space while an MR user remains perceptually and functionally connected to the surrounding physical environment (Olin et al., 2020; Huang et al., 2024). At the same time, differences in immersion, embodiment, environmental awareness, and interaction capability may create imbalances in how partners perceive, coordinate with, and respond to one another. Prior work on asymmetric XR collaboration suggests that device type, spatial layout, and avatar representation can shape social presence and interaction behavior (Shin et al., 2022; Yoon et al., 2023; Qiu et al., 2023), but it remains unclear how these differences affect social touch. The central question then is not simply whether haptic feedback can be delivered in XR, but whether the social meaning of touch can be communicated and experienced when the interaction is mediated through VR, MR, or a split reality between the two.

In this work, we explore real-time social touch across VR, MR, and cross-reality VR–MR interaction. We present a shared multi-user XR system in which physically distant users can initiate and receive touch through avatars while receiving spatially distributed vibrotactile feedback on the hands and forearms (Figure 1). We evaluate how effectively emotions and affective touch intentions can be communicated across real, virtual, mixed, and cross-reality conditions, and how social touch in VR and MR shapes embodiment, social presence, usability, haptic experience, and user preference. Through this, we examine not only whether mediated touch can be felt in XR, but whether social touch can remain meaningful when users are embodied across different realities.

2 Background and related work

Mixed Reality (MR) experiences that embed virtual humans and objects into the physical world, offer a distinct setting for mediated social touch because users remain perceptually anchored to their own body, surrounding objects, and local space while interacting with virtual entities. Recent XR haptics work has framed social interaction as a major challenge and opportunity for haptic feedback across virtual, augmented, and mixed realities (Pacchierotti et al., 2024), while work on embodiment in MR has emphasized that avatar-related experiences in see-through settings must be understood in relation to the simultaneously visible physical body and environment (Genay, 2022). Prior studies in Augmented Reality (AR) have shown that virtual agents can meaningfully affect social behavior, like influence task performance, nonverbal behavior, and social responses (Miller et al., 2019; Jun and Bailenson, 2020). These findings suggest that socially meaningful real–virtual encounters can emerge in MR settings, making them a promising foundation for future mediated touch systems.

2.1 Haptic interfaces for mediated social touch

Physical–virtual coherence and multisensory coupling in social encounters have been explored in several MR studies. Kim et al. showed that maintaining a virtual human’s spatial and behavioral coherence with physical objects increases social presence and supports more plausible interaction with a virtual agent (Kim et al., 2017). Lee et al. investigated proxemics with virtual humans in AR and found that vibrotactile cues can partly compensate for limitations of optical see-through displays, with benefits appearing more clearly in subjective co-presence than in objective locomotion behavior (Lee et al., 2018a). Related work has also shown that airflow synchronized with a virtual human can improve co-presence in AR (Kim et al., 2018), and that a virtual human appearing to exert physical influence on a shared tabletop object increases co-presence and perceived physicality (Lee et al., 2018b). Thus, MR social experiences benefit from stronger coupling between virtual humans and the physical world, as well as from carefully designed multisensory feedback.

On one hand, MR-oriented haptic interfaces have aimed to “augment” direct interaction with the physical world. Previous work has introduced wearable fingerpad-free haptic devices with side-of-finger and fingernail vibrotactile actuation that leaves the fingertips free for manipulating real objects while providing haptic feedback for hand-based virtual interaction in MR/AR (Preechayasomboon and Rombokas, 2021; Maeda et al., 2022; Hulin et al., 2020; Teng et al., 2021). These devices represent important enabling steps for hand-based haptics in AR/MR, but they are limited to interactions at the fingertip, therefore primarily target object interaction, textures, tools, and manual dexterity instead of interpersonal social touch between remote users.

Efforts to develop haptic devices for mediated social touch in VR/AR/MR have explored a wide range of mechanisms, body locations, and application goals, including social touch communication, affective messaging, and enhanced multimedia experiences (Arafsha et al., 2015; Akshita et al., 2015; Mullenbach et al., 2014; Yoo et al., 2015; Chandra et al., 2018; Israr et al., 2014; Rognon et al., 2022). Many of these systems target the forearm, both because it is well suited for actuator integration and because it is a socially appropriate site for interpersonal touch (Suvilehto et al., 2015). Prior work has employed modalities such as vibration (Huisman et al., 2016; Huisman et al., 2013; Cabibihan et al., 2012; Cang et al., 2023; Kirchner et al., 2023; Richard et al., 2025), slow or static pressure (Muthukumarana et al., 2020), force (McLaughlin et al., 2008; Wang et al., 2012), and temperature (He et al., 2024; Hecquard et al., 2024), each affording different tradeoffs in realism and affective expressiveness. Likewise, actuation strategies including linear actuators (Ferguson et al., 2022), pneumatics (Wu and Culbertson, 2019), voice-coil actuators (Zhu et al., 2022; Culbertson et al., 2018), and mid-air stimulation (He et al., 2024) vary in their implications for wearability, scalability, and fidelity. Among these, vibration-based tactile illusions (Lederman and Jones, 2011; Schneider et al., 2015; Israr and Poupyrev, 2011; Meyer et al., 2016) are especially versatile, as they can evoke continuous and spatially distributed motion patterns from compact, wearable hardware, making them promising for rendering social touch gestures. Despite this rich body of hardware research, comparatively few studies have evaluated such systems within actual interactive VR/MR scenarios involving real-time, bidirectional, human-to-human social touch, where haptic feedback is provided to both the initiator and the receiver. To date, this remains largely underexplored, with only a small number of very recent systems beginning to address it (Banerjee et al., 2025; Tong et al., 2024).

2.2 Social touch in mixed reality

Prior work approaching interpersonal social touch more directly in MR has largely remained within human-agent rather than human-human interaction. Kotranza and Lok’s Mixed Reality Humans co-registered a virtual human with a tangible physical proxy, enabling touch-based interaction in a clinical training scenario. Their evaluation demonstrated usability, acceptability, and support for interpersonal-touch behaviors with a virtual patient (Kotranza and Lok, 2008; Kotranza et al., 2009). Nonetheless, this paradigm was local, centered on human-agent interaction, and oriented toward clinical examination rather than remote, reciprocal social touch. More recent distributed MR work strengthens the case for asymmetric, cross-platform interaction, but still stops short of mediated touch. Shin et al. studied a technologically asymmetric collaboration task with one user in AR and one in VR, showing that device type and spatial layout affect co-presence and social presence (Shin et al., 2022). Yoon et al. further showed that avatar visibility in task-centric MR collaboration should be tuned to context, with AR users preferring more vivid avatars than VR users even when strong visual presence is not always required for task completion (Yoon et al., 2023). ViGather integrated traditional screen-device users and MR-headset users into a shared avatar-based conferencing environment that supports nonverbal cues and improves several presence-related measures for screen-device users (Qiu et al., 2023). Overall, cross-reality social interaction depends strongly on how avatar appearance, attention, and body-related cues are represented across platforms.

In VR, social touch has been studied across virtual-human, avatar-mediated, and human-human settings, and across both user-received and user-initiated touch (Jacucci et al., 2024). Within virtual-human and avatar-mediated interaction, prior work has shown that social touch can increase perceived human-likeness and co-presence, while responses vary with touch force, partner cues, and body region (Hoppe et al., 2020; Bailenson and Yee, 2008). Received virtual touch has also been examined in relation to body image, body location, erogeneity, and individual differences, showing realistic but context- and user-dependent affective responses (Tremblay et al., 2016; Fusaro et al., 2021; Harjunen et al., 2017). Virtual touch and other nonverbal cues shape persuasion, compliance, and behavioral responses, although their effects are often task- and cue-dependent rather than uniformly robust (Swidrak and Pochwatko, 2019; Harjunen et al., 2018; Zhao et al., 2018). More recently, mediated-touch work has considered the touch initiator, showing that initiator-side haptic feedback can affect co-presence, pleasantness, and stroking behavior, but the study was mainly within constrained stroking interactions (Maunsbach et al., 2023), and not free exploration. In human-human VR and telepresence settings, prior work has explored remote handshake and force-feedback gestures, showing benefits for telepresence or social-intention communication (Kunii and Hashimoto, 1995; Hashimoto and Manoratkul, 1996; Ouchi and Hashimoto, 1997; Nakanishi et al., 2014; Pedemonte et al., 2016; Pedemonte et al., 2017; Tong et al., 2024; Alhalabi and Horiguchi, 2001; Miyoshi et al., 2015). Multi-user remote VR+haptic social-touch systems and experiences have shown that virtual touch can evoke emotion patterns shaped by intimacy, touch direction, and user factors, but they remain largely VR-only or limited in scale or gesture scope (Banerjee et al., 2025; Sykownik and Masuch, 2020; Kimmel et al., 2026). Broader affective vibrotactile VR experiences likewise show that haptics can increase arousal, illusion of touch, or social presence, but often rely on scripted triggers or visual–haptic comparisons rather than live reciprocal touch (Bourdin et al., 2013; Peng et al., 2024; Tietenberg et al., 2026).

In contrast to VR, prior MR work has mainly investigated social influence from virtual humans, physical–virtual coherence, avatar representation, and haptic hardware, alongside a smaller body of human-agent touch systems. To the best of our knowledge, no MR system has demonstrated live, real-time, human-human mediated bidirectional social touch, where both physically distant users can communicate using touch and be touched while both receive spatially distributed haptic feedback. More specifically, expressive interpersonal touch has not been studied in a bidirectional MR framework, let alone in a cross-reality human-human bidirectional social touch setting where one user participates from VR and the other from MR. This configuration is especially important because it combines two different perceptual anchors: the VR user is fully embedded in a synthetic environment, whereas the MR user remains situated in the physical world. Prior asymmetric MR collaboration work (Shin et al., 2022; Yoon et al., 2023; Qiu et al., 2023) suggests that these two sides will not experience the shared interaction in the same way, making cross-reality social touch not only a compelling application scenario, but also a scientifically important testbed for studying how touch intentions, affect, plausibility, and other user behaviors transfer across levels of immersion.

3 System design

3.1 Multiplayer VR–MR environment

Our system provides a shared networked environment for real-time social touch across both VR and MR. It supports users interacting with avatars, virtual objects, and spatially anchored content while receiving synchronized haptic feedback. The system builds on our prior multi-user VR platform for real-time social touch (Banerjee et al., 2025), extending it beyond fully virtual interaction to support MR and cross-reality use cases. We treat VR and MR as different presentation modes of the same shared networked space. The underlying multiplayer architecture, avatar synchronization, collision detection, object interaction, and haptic communication pipeline remain common across both realities, while each client locally determines whether the environment is rendered as a fully virtual scene or overlaid onto the user’s physical surroundings.

The application was developed in Unity for Meta Quest headsets and supports both hand tracking and controller-based input. Real-time multi-user audio-visual communication and synchronization of avatars and interactable objects are handled through Photon Fusion, allowing users in different physical locations to interact within the same session. At the start of each session, users can select their own avatar, which is then instantiated and synchronized across the shared environment. The system runs natively on Quest devices and can also operate through Quest Link or Air Link. Users can move through natural locomotion within their tracked play area or through joystick-based navigation when physical space is limited. The current implementation supports up to 16 concurrent users in a single session.

In the VR configuration, users are placed in a shared virtual room containing a central table and multiple manipulable objects. These objects can be used either collaboratively or independently, while avatar motion, object states, and touch events are synchronized across users. In the MR configuration, passthrough rendering replaces the virtual background with the user’s real-world environment. The virtual ground plane is removed, and shared virtual elements, including avatars, tables, and interactable objects, are spatially positioned within the user’s physical space. This allows users to interact with virtual content while maintaining visual awareness of their surroundings. To support first-person usability in MR, the local avatar is instantiated with its visible mesh disabled, while its skeletal rig, tracking transforms, and collider structure remain active. This prevents the local avatar body from visually obstructing the user’s view, while still allowing the hands and interaction points to drive collision-based haptic events. Remote avatars remain fully rendered so that users can see the other participant’s body motion, gestures, and touch actions.

A key challenge in the MR configuration is preserving perceptual consistency between virtual content and real-world geometry. Without additional handling, virtual objects can appear visually layered in front of real hands or physical objects, weakening the sense that the interaction is grounded in the user’s immediate environment. To address this, we incorporate an occlusion-based rendering strategy using available depth segmentation information, allowing real-world hands and objects to occlude virtual geometry when appropriate. This improves spatial alignment between physical and virtual elements and makes avatar-mediated touch feel more naturally embedded in the surrounding space.

The same architecture supports both symmetric MR–MR interaction and asymmetric VR–MR interaction. In MR–MR sessions, both users see the other participant’s avatar and shared virtual objects embedded within their own physical environment. In VR–MR sessions, one user remains immersed in the virtual room while the other interacts through passthrough MR. In both cases, users share the same networked avatar state and receive haptic feedback through the same collision-based rendering pipeline. As a result, social touch is not tied to a single reality condition: users can initiate and receive touch whether both participants are in VR, both are in MR, or one user is in VR while the other is in MR.

3.2 Collision-based haptic rendering

We employ a collision-driven framework for rendering localized vibrotactile feedback through a distributed array of actuators mapped to avatar joints. The system represents each interaction point using spherical colliders attached to the avatar’s fingers and forearm, establishing a direct correspondence between virtual contact events and physical actuation. This design allows scalability in tactile resolution, where the number and placement of contact points can be adapted based on the capabilities of the haptic device. The glove integrates a total of 14 ERM actuators across the hand, drawing on the design methodology proposed in (Ariza et al., 2016), along with 12 additional ERMs distributed along the forearm (6 on the dorsal side and 6 on the volar side), informed by (Israr and Abnousi, 2018). This arrangement is intended to support continuous and perceptually smooth haptic feedback. The placement strategy is guided by neurophysiological considerations, specifically targeting regions dense in fast-adapting mechanoreceptors such as Pacinian corpuscles, which are particularly sensitive to high-frequency vibration in the fingers and palm. Consistent with prior work (Ariza et al., 2016), the actuator layout also reflects a balance between maintaining device mobility, minimizing power requirements, and preserving user comfort, resulting in a wearable system that remains both effective and unobtrusive.

We drive the ERM actuators using Pulse Width Modulation (PWM), the parameters of which were tuned through a pilot study with 8 participants to balance two key requirements: (1) long-term comfort under continuous exposure, and (2) sufficient perceptual range to reliably distinguish variations in intensity corresponding to different interaction cues. Our haptic rendering pipeline captures both the occurrence of contact and its magnitude. Beyond binary collision detection, we encode the extent of interaction through a Penetration Depth Information (PDI) metric. PDI is defined as the difference between the maximum allowable penetration distance and the current distance from the initial contact point (Figure 2), for the remainder of this paper. Each message sent to the microcontroller includes the actuator ID, PDI, surface normal, and a flag indicating first contact. Each sphere attached to the hand and forearm is equipped with a collider and uniquely tagged. Upon initial contact, the system transmits a PDI of 0 with the first-contact flag set to true, triggering a short, high-amplitude pulse that produces a discrete “click” sensation. This provides a clear tactile cue for contact onset. The flag is then immediately reset, and subsequent frames are treated as sustained contact.

3.2.1 Interacting with virtual objects

For continuous contact, the ERM outputs a baseline vibration (12% duty cycle for the hand, 24% for the arm). As the user presses further into the object, amplitude scales linearly with decreasing PDI, reaching a maximum (25% for the hand, 50% for the arm) at full penetration. Object stiffness is encoded via the maximum allowable penetration depth, with stiffer objects limiting displacement and thus producing steeper amplitude gradients. Once this limit is reached, the actuator maintains its peak output to convey resistance. For grasping interactions, the feedback transitions from the initial click to a gradual reduction in amplitude, stabilizing at a low baseline (12% duty cycle for the hand). This attenuation provides a sustained “holding” sensation while avoiding discomfort from prolonged high-intensity stimulation.

3.2.2 Interacting with other avatars

Each avatar is represented by sphere arrays aligned with actuator locations (Figure 3). Smaller spheres are used on the fingers for spatial precision, while larger spheres on the forearm ensure smoother coverage. During avatar-avatar interaction, vibration intensity reflects the degree of overlap. The maximum penetration threshold is fixed as the diameter of the smaller sphere. At initial contact, a minimum amplitude signal is applied (12% duty cycle for the hand, 24% for the arm), increasing linearly with overlap depth. Maximum amplitude is reached when the smaller sphere is fully embedded within the other. This results in 24% duty cycle for finger–finger interactions and up to 50% for forearm-involved contacts, ensuring consistent perceptual scaling across interaction types. When no collision is detected, no signal is transmitted, and the system resets the first-contact flag, preparing for the next interaction event.

3.3 Hardware setup and runtime pipeline

The system is designed to be fully wireless and plug-and-play, making it adaptable to any tactile device supporting WiFi and/or serial communication. For our wearable setup, we use Arduino Uno R4 WiFi microcontrollers, with one dedicated to each hand, and both connected to the same network as the VR headset. Each hand is equipped with a haptic glove and forearm sleeve, driven by an Arduino Uno R4 WiFi and two stacked 16-channel 12-bit PWM drivers (Adafruit PCA9685; Figure 3B). The entire system is powered by a 5V 10000 mAh battery pack, housed alongside the electronics in a running armband, enabling a fully portable and untethered setup without reliance on external power or computation. The gloves and sleeves are made from breathable spandex to ensure comfort and accommodate a range of hand sizes. Each microcontroller-PWM stack drives 5V, 11000 RPM ERMs (Adafruit 1528-1177-ND) embedded within the gloves and sleeves. ERMs were chosen for their low latency, small form factor, and simplicity of operation in real-time applications. These qualities allow for VR compatibility, ease of integration into wearable designs, and perceptible haptic feedback within a usable range of frequencies and amplitudes. Each ERM is connected to the PWM shield via a transistor, which controls the amount of current flow to generate a wide range of vibrotactile sensations.

Each microcontroller identifies itself over WiFi by sending an initial User Datagram Protocol (UDP) packet containing its role (“left” or “right” hand) and local IP address. The Unity application broadcasts a discovery message and receives these responses, after which it establishes a dedicated Transmission Control Protocol (TCP) connection with each device. TCP was chosen over UDP to ensure reliable packet delivery. Each transmitted packet is at most 21 bytes, well within the Arduino’s Maximum Transmission Unit (MTU), resulting in negligible communication overhead and low latency. The system operates at 100–120 Hz, with Unity running at 120 FPS and updating the Penetration Depth Information (PDI) every frame. All signal processing, including amplitude and frequency computation, is performed locally on the microcontroller. Rather than transmitting actuator commands directly, we send encoded collision data (including actuator ID, PDI, contact normal, and contact state), allowing various devices to interpret the interaction based on their actuation modalities. This design supports a device-agnostic framework: while our system maps this data to vibrotactile output, other devices (e.g., force, pressure, or ultrasonic systems) can leverage the same information differently. In our implementation, contact normals are transmitted but not utilized, as the device provides only vibrotactile feedback.

4 Experimental evaluation

4.1 Participants

Pairs of participants completed the study together. We recruited 44 participants in total. Data from two participant pairs were excluded: one session was interrupted because of hardware issues, and one pair discontinued the study midway due to fatigue. The final dataset therefore included 40 participants, forming 20 dyads (26 females, 14 males; M age = 26.78 ± 2.75 years), of whom 8 reported prior experience with haptic devices and 20 reported prior experience with VR. There were 10 male-female dyads, 2 male-male dyads, and 8 female-female dyads. All participants knew their partner beforehand, except for one male-male dyad. None of the participants reported sensory or motor impairments. The study was approved by the University of Southern California Institutional Review Board under protocol UP-19-00712, and all participants provided informed consent. During the experiment, participants wore noise-cancelling earphones connected to the VR headset, which both delivered audio from the simulation and helped mask noise produced by the ERMs.

4.2 Procedure

We conducted a two-part within-subjects study consisting of Prescribed Touch and Free Exploration. The order of the two experiments, as well as the conditions within each experiment, was counterbalanced to minimize potential order effects. To reduce fatigue associated with the length of the study, the two experiments were conducted in separate sessions, with an inter-study interval of at least 4 to at most 48 h.

4.2.1 Prescribed touch

The Prescribed Touch task evaluated how effectively participants could communicate affective intent through touch across real, VR, MR, and cross-reality interactions. Each dyad completed 5 reality conditions counterbalanced: real–real (RR–RR), virtual–virtual (VR–VR), virtual–mixed (VR–MR), mixed–virtual (MR–VR), and mixed–mixed (MR–MR) realities. In RR–RR, both participants interacted through direct physical touch in a real physical setting. They sat side by side with an unobstructed view of one another but were instructed to avoid eye contact and not look at the touch initiator during touch interactions. In the remaining conditions, participants interacted through the corresponding VR/MR system configuration. For these mediated conditions, participants were physically separated into two rooms and interacted exclusively through the networked XR environment. Their tracked head and hand movements animated their corresponding avatars within a shared coordinate space, where both avatars were positioned in front of a virtual table. VR users experienced the interaction entirely within the shared virtual environment, whereas MR users viewed the remote participant’s avatar overlaid onto the passthrough view of their physical environment. Touchers were explicitly instructed to convey emotions exclusively through touch gestures, and to avoid deliberately conveying it through speech, facial expressions or other non-touch signals. Within each reality pairing condition, participants alternated between the role of initiator and receiver in a counterbalanced order. In each trial, the initiator was prompted to convey one of seven emotions—Love, Happiness, Anger, Sadness, Gratitude, Calm, and Attention—using touch limited to the receiver’s hands and forearms. We selected these based on prior human-to-human and machine-rendered social-touch research (Hauser et al., 2019; Salvato et al., 2021; McIntyre et al., 2022), spanning positive affiliative, supportive, and attention-directing touch messages. In our study, anger was newly added as a negatively valenced, high-arousal emotion because prior work has shown that it can be communicated reliably through touch and is generally associated with more forceful and dynamic touch behaviors (Hertenstein et al., 2009). The prompts for each emotion were delivered only to the initiator through an audio recording, which began by stating the target emotion so participants could voluntarily skip the remainder of the recording during repeated trials in later modality conditions. Receivers did not hear the prompt. After conveying the emotion, the initiator indicated trial completion by either returning their hand to their lap during RR–RR interactions or pressing a “Done” button in the VR/MR conditions. The receiver then reported their interpretation of the conveyed emotion by assigning probabilities to each of the seven emotion categories using sliders. This probability-based response format allowed receivers to express uncertainty or mixed interpretations rather than selecting a single categorical label. Each participant conveyed each emotion once per modality pairing. Thus, each pair completed 14 trials per modality pairing and 70 trials across the five modality pairings. The experiment lasted approximately 2 h per participant pair, including a 10 min break in between reality conditions.

4.2.1.1 Experimental measures

We collected both objective and subjective measures. For VR and MR interactions, we recorded the initiator’s hand trajectory, including 3D position and velocity, within the virtual environment. For RR–RR interactions, hand motion was recorded using an Intel RealSense Depth Camera D435i and processed using MediaPipe Hands, which estimates a hand skeleton from single-camera input using a palm detector followed by a hand-landmark model (Zhang et al., 2020). We also extracted gesture dynamics from the motion data by classifying interactions into five fundamental touch gestures–stroke, squeeze, tap, pat, and poke–which could be consistently identified and rendered across the real, VR, MR, and cross-reality conditions using our wearable vibrotactile interface. For stroking gestures, we computed the stroking velocity in cm/s. For all other gestures, we computed gesture frequency by averaging the number of repeated gesture events over the time interval in which they were performed. Subjective measures included trial-level Valence and Arousal ratings from both initiators and receivers using the EmojiGrid (Toet and Van Erp, 2019). To monitor cybersickness, participants completed the Simulator Sickness Questionnaire (SSQ) (Kennedy et al., 1993) before and after each reality condition involving VR or MR.

4.2.2 Free exploration

The Free Exploration task evaluated how participants used the system during unconstrained social touch interaction. Participants were first instructed on how to wear the gloves and sleeves and how to launch the VR/MR application on the Meta Quest headset. They were then asked to don the hardware and start the application independently, without further assistance. After entering the environment, participants were instructed to interact freely with each other. The only requirement was that they find a way to engage in touch-based interaction using their hands and/or lower arms. If direct touch felt uncomfortable or unnatural, participants were encouraged to create a shared gesture, such as a secret handshake, involving the hands and/or lower arms. Beyond this requirement, participants were free to interact with virtual objects or collaboratively perform any task of their choice. The task was conducted in the VR–VR and MR–MR conditions, counterbalanced across participant pairs. The Free Exploration phase was intentionally restricted to the symmetric VR–VR and MR–MR configurations to isolate experiential and behavioral differences between matched-reality environments. Unlike Prescribed Touch, Free Exploration had no directed initiator and receiver roles. Therefore, VR–MR and MR–VR would represent the same asymmetric dyadic configuration rather than two distinct directional conditions. Open-ended cross-reality interaction was therefore outside the scope of this phase.

Each free exploration session lasted up to 10 min, with a 5-min break between reality conditions, resulting in a total duration of approximately 25 min.

4.2.2.1 Experimental measures

To investigate the relationship between participants’ real-world comfort with interpersonal touch and their experience of virtual social body contact, we administered the Comfort with Interpersonal Touch (CIT) questionnaire (Webb and Peck, 2015) prior to the experiment. After each free exploration condition, participants completed questionnaires assessing embodiment, presence, usability, haptic experience, and social presence. Embodiment was measured using the Avatar Embodiment Questionnaire (Peck and Gonzalez-Franco, 2021). Presence was assessed using the Slater-Usoh-Steed Presence Questionnaire (Slater et al., 1998; Usoh et al., 1999), rephrased for VR/MR. Cybersickness was monitored using the Simulator Sickness Questionnaire (SSQ) (Kennedy et al., 1993) filled before and after each modality condition. Perceived usability was measured using the System Usability Scale (Brooke, 1996). Participants also completed the Haptic Experience Inventory (HXI) (Shi and Schneider, 2025) and the Networked Minds Social Presence Inventory (NMSPI) (Harms et al., 2004) to assess perceived social presence across VR and MR. At the end of the study, participants indicated whether they preferred the VR or MR condition. Objective measures included touch-gesture type, gesture frequency, and gesture duration in each reality condition.

4.3 Results

4.3.1 Prescribed touch

4.3.1.1 Emotion recognition accuracy (ERA) across modality condition combinations

Confusion matrices, with row-normalized percentages, precision, recall, and overall accuracy, are presented in Figure 4. Each receiver assigned a probability to each of the seven emotions according to what they thought the received touch belonged to. The top probability was chosen as the emotion recognized corresponding to the emotion conveyed. Emotion Recognition Accuracy (ERA) was analyzed as a binary trial-level outcome indicating whether the receiver’s selected emotion matched the conveyed emotion. Emotion conveyed was also included as a within-subject factor in both analyses. No order effect was observed across the analyses. Because ERA was binary, all inferential analyses used trial-level logistic generalized estimating equation (GEE) models. Significant reality-by-emotion interactions were followed using contrasts derived from the same fitted GEE models, comparing reality conditions separately within each emotion. Holm correction was applied across all within-emotion pairwise contrasts.

For the same reality analysis (which compared RR–RR, VR–VR, and MR–MR interactions), the GEE revealed significant main effects of condition and conveyed emotion, as well as a significant Condition × Emotion interaction (Table 1). Follow-up contrasts indicated that the effect of condition was limited to Anger. Specifically, ERA was higher in the RR–RR condition than in both the VR–VR (O R = 17.00, 95% CI [5.28, 54.70], p Holm < . 001) and MR–MR (O R = 39.67, 95% CI [12.29, 128.03], p Holm < . 001) conditions. There was no significant difference between the VR–VR and MR–MR conditions after Holm correction, and no other conveyed emotion showed significant differences between conditions.

TABLE 1. Logistic GEE omnibus effects for emotion recognition accuracy.

Effect sizes for significant follow-up contrasts are reported as odds ratios with 95% confidence intervals in the text.

For the same- and cross-reality analysis (which compared interactions across VR–VR, MR–MR, VR–MR, and MR–VR), condition, conveyed emotion, and their interaction were significant. Follow-up comparisons revealed only one significant difference between conditions after correcting for multiple comparisons. Specifically, participants recognized Happiness with higher accuracy in MR–VR than in VR–MR (O R = 7.43, 95% CI [2.67, 20.69], p Holm = . 005). No other condition contrast remained significant after correction.

As a planned structure-level comparison, trials were collapsed into same-reality interactions (VR–VR and MR–MR), and cross-reality interactions (VR–MR and MR–VR). The main effect of structure was not significant, but the structure × emotion interaction was significant. However, none of the seven emotion-specific same-versus cross-reality contrasts survived Holm correction.

To test whether the receiver’s reality influenced ERA, trials were collapsed by receiver reality. Receiver-in-VR trials included VR–VR and MR–VR, whereas Receiver-in-MR trials included VR–MR and MR–MR. The GEE showed a significant main effect of receiver reality and a significant receiver reality × emotion interaction. Emotion-specific contrasts showed higher ERA when the receiver was in VR rather than MR for Gratitude (O R = 2.67, 95% CI [1.64, 4.35], p Holm < . 001), and Happiness (O R = 3.44, 95% CI [1.67, 7.09], p Holm = . 005). No other emotion-specific receiver-reality contrast was significant after correction.

Similarly, for initiator-side reality, the main effect of initiator reality was not significant, although the initiator reality × emotion interaction reached significance. None of the seven emotion-specific initiator-reality contrasts survived Holm correction; therefore, no individual emotion can be identified as showing a reliable initiator-reality difference.

The confusion matrices and ERA describe whether the receiver’s highest-probability response matched the conveyed emotion, but they do not show how the receiver’s responses distributed across the remaining emotion categories. We therefore used the probability assignments to each of the emotions to examine whether the touch produced recurring perceptual profiles, whether these profiles corresponded to the intended emotions, and whether recognition errors reflected systematic similarity between affectively related emotions across realities. This probability-assignment cluster analysis helps determine whether receivers retained the broader affective meaning of a touch even when they did not identify its precise categorical label. To examine whether participants showed consistent probability-assignment profiles beyond their top-choice responses, each trial was represented as a 7-dimensional probability vector corresponding to attention, gratitude, happiness, calm, love, sadness, and anger, similar to the analysis in (Salvato et al., 2021). K-means clustering was applied to these vectors; the seven-cluster solution (Figure 5) provided the clearest separation of response profiles, with the highest Silhouette, highest Calinski–Harabasz, and lowest Davies–Bouldin scores. The resulting clusters largely corresponded to distinct perceived-emotion profiles, as shown in Figure 5. Cluster 1 ( n = 307 ) was characterized by attention assignments ( M = . 39 ) and primarily contained trials in which attention was conveyed, with some overlap from anger. Cluster 2 ( n = 176 ) reflected a gratitude profile ( M = . 35 ), with secondary overlap from happiness and love. Cluster 3 ( n = 226 ) showed a clear happiness profile ( M = . 39 ). Cluster 4 ( n = 213 ) was dominated by calm assignments ( M = . 38 ), but also contained love and sadness trials. Cluster 5 ( n = 245 ) reflected a love profile ( M = . 36 ), with substantial overlap from calm and sadness. Cluster 6 ( n = 131 ) was dominated by sadness assignments ( M = . 38 ), again showing overlap with calm and love. Cluster 7 ( n = 102 ) reflected an anger profile ( M = . 39 ), with some attention overlap.

4.3.1.2 Analyses of gestures performed by conveyed emotion

To understand how participants expressed different affective intentions through touch, we analyzed the gestures used to convey each emotion in each reality condition. We first examined whether conveyed emotions differed in their overall gesture-component profiles across the five gesture components: stroke, squeeze, tap, pat, and poke. To account for repeated observations within dyads, gesture counts were aggregated at the dyad level for each conveyed emotion and normalized to obtain gesture-component proportions. A repeated-measures MANOVA showed that gesture profiles differed significantly by conveyed emotion (Pillai’s V = 3.166, F ( 24,456 ) = 72.11, p < . 001).

Love was primarily conveyed using squeeze and stroke, with infrequent taps and pats; Happiness using tap, pat, and squeeze; Anger using tap, pat, poke, and squeeze; Sadness using stroke and squeeze, with occasional pat components; Gratitude using pat, squeeze, and stroke; Calming using stroke and squeeze; and Attention using tap and poke (Figure 6). To quantify the concentration of each emotion’s gesture profile, normalized Shannon entropy was computed over the five gestures, with lower values indicating a more concentrated gesture repertoire and higher values indicating a broader distribution across gesture types. Attention showed the most concentrated gesture profile ( H = . 47 ), followed by Calming ( H = . 50 ), Sadness ( H = . 52 ), Love ( H = . 53 ), Happiness ( H = . 71 ), Gratitude ( H = . 76 ), and Anger ( H = . 85 ).

We then tested whether the gesture distribution differed across Initiator reality within each conveyed emotion. Gesture-component proportions were calculated at the dyad level for each initiator reality and analyzed using repeated-measures MANOVAs. After Holm correction across the seven conveyed emotions, gesture distributions differed by initiator reality only for Attention (Pillai’s V = . 563, F ( 8,72 ) = 3.53, p Holm = . 012), and were not significant for the other conveyed emotions. Holm-corrected pairwise post-hoc tests showed significant differences between RR and VR ( p Holm = . 010 ) and between VR and MR ( p Holm = . 008 ) only. To convey Attention, Real interactions contained relatively more pokes and fewer taps, VR interactions contained relatively more strokes and fewer squeezes, and MR interactions contained relatively more taps and squeezes but fewer pokes.

Gesture duration was analyzed using participant-level Friedman tests across RR, VR, and MR. The effect of initiator reality on gesture duration was significant only for Gratitude (χ 2 ( 2 ) = 6.65, p = . 036, W = . 08); however, no Holm-corrected pairwise post-hoc comparison was significant. Duration did not differ by reality for other conveyed emotions.

For gesture-specific dynamics, stroke was analyzed using stroke speed, whereas squeeze, tap, pat, and poke were analyzed using frequency. Friedman tests showed that the only significant reality effect was observed for squeeze frequency when conveying Happiness (χ 2 ( 2 ) = 10.63, p = . 005, W = . 41); pat frequency when conveying Happiness (χ 2 ( 2 ) = 7.40, p = . 025, W = . 09); and squeeze frequency when conveying Sadness (χ 2 ( 2 ) = 7.30, p = . 026, W = . 16). Holm-corrected post-hoc tests showed lower squeeze frequency in RR than VR for Happiness (W = 7.00, p Holm = . 014, r = − . 85) and lower pat frequency in RR than MR for Happiness (W = 230.50, p Holm = . 047, r = − . 44); no Holm-corrected pairwise comparison was significant for Sadness squeeze frequency. No other stroke-speed or non-stroke gesture frequency comparisons showed significant reality effects.

Finally, to assess whether the gesture features were sufficient to distinguish conveyed emotions, we trained a multinomial logistic-regression classifier using component-level gesture use, gesture counts, duration, stroke speed, and non-stroke gesture frequencies. In stratified 10-fold cross-validation, the model classified conveyed emotion with an accuracy of 78.57 % ( S D = 1.99 % ) and macro-F1 of 0.78 ( S D = . 02 ), compared with a chance level of 14.29 %. A stricter leave-one-initiator-out validation produced a mean accuracy of 78.07 % ( S D = 6.89 % ) and macro-F1 of 0.78 ( S D = . 07 ). Cross-reality generalization remained well above chance, although performance was lower than in stratified cross-validation: models trained on one initiator reality and tested on another achieved accuracies between 70.89 % and 78.57 % and macro-F1 scores between 0.71 and 0.79. Across the confusion matrices, Anger and Attention were classified almost perfectly, whereas most errors occurred among Love, Sadness, and Calming, which shared gesture profiles.

4.3.1.3 Valence–arousal mismatch and affective alignment

Receiver–initiator affective mismatch was analyzed using absolute valence and arousal differences (Figure 7). Two repeated-measures ANOVAs were then conducted: one for same reality interactions (RR–RR, VR–VR, MR–MR) and one for same-/cross-reality interactions (VR–VR, MR–MR, VR–MR, MR–VR), with condition and conveyed emotion as within-dyad factors.

The same reality analysis showed significant effects of condition and conveyed emotion for both valence and arousal mismatch, as well as significant Condition × Emotion interactions (Table 2). Holm-corrected post-hoc comparisons showed that RR–RR produced lower mismatch than both mediated same-reality conditions; VR–VR and MR–MR did not differ reliably. In the same-/cross-reality analysis, condition, emotion, and their interaction were significant for valence mismatch. For arousal mismatch, the condition main effect was not significant, but emotion and the Condition × Emotion interaction were significant. Significant condition comparisons are summarized in Table 3; non-significant comparisons are not reported. Emotion-specific post-hoc comparisons were conducted only for significant interactions (Table 4). In the same reality analysis, lower mismatch in RR–RR was seen than in VR–VR and MR–MR. In the mediated/cross-mediated analysis, the significant differences were limited to Happiness for valence mismatch and Anger for arousal mismatch.

TABLE 2. Receiver–initiator valence and arousal mismatch: significant omnibus effects.

TABLE 3. Receiver–initiator valence and arousal mismatch: significant Holm-corrected condition comparisons.

TABLE 4. Significant Holm-corrected emotion-specific simple comparisons for receiver–initiator mismatch.

Spearman correlations showed significant initiator–receiver alignment in all conditions for both valence and arousal (Table 5). Alignment was strongest in RR–RR for valence ( ρ = . 982 ) and in RR–RR for arousal ( ρ = . 970 ), indicating that direct physical touch preserved the closest affective coupling.

TABLE 5. Condition-wise Spearman alignment between initiator and receiver valence/arousal.

Receiver-side recognition certainty was negatively associated with affective mismatch. Thus, greater receiver certainty was generally associated with closer alignment between the receiver’s valence–arousal ratings and the initiator’s intended affective coordinates. Across all reality conditions, the probability assigned to the recognized emotion was negatively correlated with both valence mismatch (ρ = − . 288, 95% CI [ − . 355 , − . 219 ], p < . 001) and arousal mismatch (ρ = − . 197, 95% CI [ − . 267 , − . 125 ], p < . 001). At the individual reality-condition level, recognition probability was negatively correlated with arousal mismatch in VR–VR (ρ = − . 273, 95% CI [ − . 420 , − . 112 ], p = . 011) and with valence mismatch in MR–VR (ρ = − . 244, 95% CI [ − . 394 , − . 082 ], p = . 032).

To determine whether receiver reality altered the relationships between gesture dynamics and perceived valence or arousal, we first calculated separate Spearman correlations for receivers in RR, VR, and MR, and then tested the interaction between gesture dynamics and receiver reality using rank-based linear mixed-effects models (Table 6). Receiver reality significantly moderated the associations of stroke speed, squeeze frequency, and poke frequency with receiver valence. The stroke–valence and squeeze–valence associations were stronger in RR than in both VR and MR, with no differences between VR and MR. For poke frequency, the association differed only between RR and MR. Receiver reality did not significantly moderate the associations of tap or pat frequency with valence. For arousal, faster or more frequent gesture dynamics generally showed positive associations across receiver realities. However, none of the gesture-dynamics-by-receiver-reality interactions for arousal were significant after Holm correction. Thus, receiver reality influenced how stroke speed, squeeze frequency, and poke frequency related to receiver valence, but there was no evidence that it altered the relationship between gesture dynamics and receiver arousal.

TABLE 6. Receiver-reality-specific Spearman correlations between gesture dynamics and receiver affect across gesture-occurrence trials.

For each gesture and receiver reality, n represents the number of trials in which the corresponding gesture occurred. Stroke dynamics were measured as speed in cm/s, whereas squeeze, tap, pat, and poke dynamics were measured as frequency per second. Gesture-absence trials were excluded. The 95% confidence intervals were calculated using the Fisher z transformation. The p int,Holm column reports Holm-corrected interaction tests from rank-based linear mixed-effects models containing gesture dynamics, receiver reality, and their interaction.

4.3.1.4 Receiver valence and arousal in mediated-only interactions

To test whether receiver-side affective ratings differed across reality conditions, RR–RR trials were excluded and a 2 × 2 × 7 repeated-measures ANOVA was conducted with initiator reality (VR, MR), receiver reality (VR, MR), and Emotion Conveyed as the within-dyad factors. Receiver affective ratings were dominated by conveyed emotion rather than by a simple main effect of initiator or receiver reality (Table 7). Overall, receiver affective ratings did not show a general VR-vs-MR difference; instead, reality effects appeared selectively for specific emotions. Significant Holm-corrected post hoc comparisons were limited to Happiness for receiver valence (VR–VR>VR–MR, t ( 19 ) = 4.81 , p = . 005 , d z = 1.08; MR–MR>VR–MR, t ( 19 ) = 4.62 , p = . 008 , d z = 1.03), and Anger for receiver arousal (VR–VR>VR–MR, ( t ( 19 ) = 4.52 , p = 0.01 , d z = 1.01 ).

TABLE 7. Significant effects for receiver valence and arousal in mediated-only interactions.

4.3.1.5 Simulator sickness for interactions in VR and MR

Shapiro–Wilk tests confirmed that the data was not normally distributed. Hence, Wilcoxon signed-rank tests were conducted to compare p o s t − p r e SSQ scores between VR and MR across participants (Figure 8A). Nausea scores differed significantly between realities (W = 104.50, p = . 039, r = . 55). Oculomotor scores also differed significantly (W = 163.50, p = . 006, r = . 59). The difference in disorientation scores was not statistically significant ( p > . 05 ). Total SSQ scores differed significantly between realities (W = 118.50, p = . 002, r = . 65). Overall, lower scores were observed in VR than in MR. After Holm-corrected adjustments, the oculomotor difference remained significant ( p = . 018 ) as did the difference in total SSQ scores, ( p = . 007 ) but the nausea difference was no longer significant. Friedman tests to compare SSQ scores across the same-/cross-reality combinations and direct comparisons between same-reality and cross-reality conditions revealed no significant differences across any SSQ outcome.

4.3.2 Free exploration

4.3.2.1 Avatar embodiment questionnaire

After checking for normality using Shapiro–Wilk tests, Wilcoxon signed-rank tests were used for non-normal difference scores, and a paired-samples t-test was used for normally distributed difference scores when comparing between VR and MR for each of the subscales of the Avatar Embodiment Questionnaire (Figure 8D). Order effects were tested by comparing the difference scores between MR and VR across the order in which participants experienced the realities and the order in which participants did the Free Exploration experiment. Appearance and Response did not differ significantly between VR and MR. Ownership (W = 9.00, p < . 001, r = . 964), Multi-sensory (W = 10.00, p = . 034, r = . 430) and total Embodiment (t ( 39 ) = 4.49, p < . 001, d z = . 71) were significantly higher in MR than in VR. For the significant effects, there was no order effect. However, Appearance showed a significant order dependency (W = 20.00, p = 0.037, r = 0.619), and was higher in MR than VR for participants who did the Free Exploration experiment later.

4.3.2.2 Haptic experience inventory (HXI)

HXI questionnaire scores were compared between VR and MR using participant-level paired tests (Figure 8E). Wilcoxon signed-rank tests were used for non-normal difference scores, and paired-samples t-tests were used otherwise. Autotelics was significantly higher in MR than in VR (W = 76.50, p = . 032, r = . 490). Involvement was significantly higher in VR than in MR (W = 120.00, p = . 030, r = − . 448). Realism, Harmony, Discord, and General Score did not differ significantly between VR and MR. No significant order effects were found for any HXI subscale or for the General Score.

4.3.2.3 Networked minds social presence inventory (NMSPI)

NMSPI scores were compared between VR and MR using participant-level paired tests (Figure 8F). After Shapiro–Wilk tests checking for normality, Wilcoxon signed-rank tests were used for non-normal difference scores, and a paired-samples t-test was used for normally distributed difference scores. Co-Presence was significantly higher in VR than in MR (W = 6.00, p < . 001, r = − . 937). Attention was significantly higher in MR than in VR (W = 54.50, p < . 001, r = . 712). Behavioral Interdependence was significantly higher in VR than in MR (W = 52.50, p = . 025, r = − . 500). Message Understanding, Affective Understanding, Emotional Interdependence, and Overall Social Presence did not differ significantly between VR and MR.

For order effects, Wilcoxon signed-rank tests showed that Affective Understanding was significantly higher in MR than VR (W = 12.00, p = . 035, r = . 636) for participants who did the Free Exploration phase first. In contrast, Affective Understanding was significantly higher in VR than MR for the other half of participants (W = 13.00, p = . 021, r = − . 667). Overall Social Presence also showed an order effect, and was significantly higher in VR than MR (W = 9.00, p = . 010, r = − . 802) for participants who did the Free Exploration phase later. No significant order effects were found for the rest of the subscales.

4.3.2.4 Slater-usoh-steed presence and simulator sickness

A Wilcoxon signed-rank test was used for Slater-Usoh-Steed Presence and SSQ responses after normality violation. The Presence scores (Figure 8B) were relatively high for both VR ( M V R = 5.63 , S D V R = 0.67 ) and MR ( M M R = 5.41 , S D M R = 0.66 ), and the simulator sickness scores were negligible with only noticeable factors being fatigue (M V R = 0.04 , S D V R = 0.19; M M R = 0.05 , S D M R = 0.11) and eyestrain (M V R = 0.15 , S D V R = 0.51; M M R = 0.13 , S D M R = 0.43). Neither Presence nor SSQ scores differed significantly between VR and MR, and there were no order effects.

4.3.2.5 System usability and participant preferences

A paired-samples t-test for System Usability showed that it was significantly higher in MR than in VR (mean difference = 4.05, 95% CI [0.65, 7.45], t ( 39 ) = 2.41, p = . 021, d z = . 38) with no significant order effects (Figure 8C).

Participant preference between VR and MR was analyzed as a post-comparison categorical choice. 28 of 40 participants preferred MR, whereas 12 preferred VR. Preference also varied by order, for participants who did the Free Exploration experiment first, 18 participants preferred MR and two preferred VR; in the other half, 10 participants preferred MR and 10 preferred VR (Fisher’s test odds ratio = 9.00, p = . 014).

4.3.2.6 Observed touch behavior and gesture dynamics

Characteristics of the free-exploration touch behavior in MR and VR are shown in Figure 9. Objective comparisons during the Free Exploration were conducted using Wilcoxon signed-rank tests and order effects were accounted for using Mann–Whitney U tests based on order of experiment as well as order of reality experienced. Holm correction was applied wherever applicable.

Participants spent more total time in VR than in MR (W = 0.00, p < . 001, r = 1.00). They also accumulated more interpersonal touch time in VR (W = 42.00, p = . 017, r = . 60) and spent a greater proportion of the session touching each other (W = 52.00, p = . 048, r = . 50). Participants were also observed to explore a broader set of touch gestures in VR than in MR (W = 0.00, p < . 001, r = 1.00). Mean gesture duration and the number of distinct interpersonal or collaborative actions explored was higher in VR than MR. Gesture-specific, Holm-corrected comparisons showed a higher tap rate in VR (W = 40.00, p = . 048, r = . 64) and a higher poke rate in VR (W = 11.00, p = . 020, r = . 94). Squeeze and pat rates did not differ significantly after correction ( p > . 05 ). Among the eleven participant pairs that performed stroking in both realities, stroke speed was higher in VR (W = 0.00, p = . 004, r = 1.00).

The order of experiments performed had a substantial overall effect on observed interaction. Participant pairs who performed the Free Exploration session first had greater total session duration (U = 90.00, p = . 017, r = . 80); cumulative touch time (U = 90.00, p = . 017, r = . 80); touch proportion (U = 88.00, p = . 017, r = . 76); touch-event count (U = 90.00, p = . 017, r = . 80); and touch-event rate (U = 90.00, p = . 017, r = . 80). There was no significant Reality × Order dependency after Holm corrections.

Order effects were also observed for exploratory behavior. Pairs who did Free Exploration experiment first used a greater variety of gestures (U = 91.50, p = . 006, r = . 83), produced longer gestures (U = 90.00, p = . 006, r = . 80), generated a higher overall gesture rate (U = 90.00, p = . 006, r = . 80), and explored more distinct actions (U = 91.50, p = . 006, r = . 83). Significant Reality × Order dependencies were found for mean gesture duration (U = 87.00, p = . 016, r = . 74), overall gesture rate (U = 89.00, p = . 014, r = . 78), and action variety (U = 80.00, p = . 017, r = . 60). Holm-corrected tests showed higher values in VR than MR within both order groups for all three measures (all p = . 004 and r = 1.00), with larger VR–MR differences among pairs who did Free Exploration first. Frequency of poke also showed an order dependency (U = 83.00, p = . 040, r = . 66): poke rate was higher in VR among pairs who did Free Exploration first (W = 0.00, p = . 016, r = 1.00), but not among the other participant pairs ( p > . 05 ).

The open-ended actions are summarized descriptively. Greeting and contact rituals, including handshakes, high-fives, fist bumps, and arm bumps, occurred frequently across both VR and MR. Collaborative virtual-object activities, including passing, holding, stacking, pulling, inspecting, and balancing, were also common in both realities. Although heavily dependent on participant pairs and order of experiments, VR produced a slightly broader range of playful or gamified activities, including synchronized clapping, tag games, hand-chasing, whereas MR interactions more often centered on hand alignment, handshakes, and guided hand movements.

4.4 Discussion

Rather than treating emotion recognition, gesture production, embodiment, and user experience as independent outcomes, we interpret our findings as describing successive stages of avatar-mediated social touch communication. A social touch interaction begins with an initiator expressing an intention through a touch gesture, which is then represented by the XR system through the avatar and haptic rendering, perceived by the receiver within a particular reality (VR or MR), and ultimately interpreted as an affective or social message. Differences between realities may therefore arise because users produce different touch behaviors, because the system renders those behaviors differently, or because identical rendered cues are interpreted differently depending on the receiver’s perceptual and embodied context.

Viewing the results through this framework helps explain several of our findings. Some aspects of social touch, such as the structure of touch gestures, remained remarkably stable across realities, whereas others, such as emotion recognition and affective alignment, depended more strongly on the receiver’s reality. Likewise, the experiential differences between VR and MR suggest that each reality shapes the context in which touch is perceived rather than simply increasing or decreasing communication quality along a single dimension. The following discussion is organized around these stages of communication, beginning with whether social touch remains meaningful across realities, then examining how gesture production and interpretation differ, before considering how VR and MR support distinct forms of embodied social interaction and what these findings imply for the design of future social touch systems.

4.4.1 Social touch remains meaningful across realities

The central finding of our experiments is that social and affective meaning of mediated touch remained structured across VR, MR, and cross-reality interaction, compared to real physical touch. Although RR–RR produced the highest overall accuracy (62.50%), followed by VR–VR (49.64%), MR–VR (46.07%), MR–MR (40.71%), and VR–MR (35.00%), more in-depth analyses revealed a considerably narrower effect of reality than these overall percentages might suggest. Direct touch outperformed the two same-mediated conditions specifically for Anger, while the only reliable difference among the four mediated pairings was higher Happiness recognition in MR–VR than in VR–MR. Moreover, collapsing the mediated conditions into same-reality and cross-reality structures produced no reliable overall difference. Participants frequently expressed surprise and enthusiasm about feeling contact from an avatar or virtual object that was not physically present. Many described the synchronized tactile feedback as “contributing substantially to immersion”. Thus, placing participants in different realities did not generally disrupt social-touch communication or create a distinct cross-reality deficit.

The probability-assignment and affective-alignment analyses further show that categorical accuracy captures only one level of what participants perceived. Attention, Happiness, and particularly Anger generated comparatively distinctive probability profiles, whereas Love, Sadness, and Calming frequently overlapped. Gratitude occupied an intermediate position, retaining a recognizable profile while sharing perceptual space with Happiness and Love. These confusions were not arbitrary: they tended to occur among emotions with related levels of arousal, valence, or affiliative meaning. Cross-mediated trials were distributed across the same clusters as VR–VR and MR–MR trials rather than forming a separate response space, suggesting that perceived affective content organized participants’ responses more strongly than whether the two users occupied matching realities. Receiver–initiator valence and arousal ratings support the same interpretation. Direct physical touch produced the closest affective coupling, with lower mismatch than the same-mediated conditions and the strongest initiator–receiver correlations. Nevertheless, significant affective alignment remained present across all reality pairings. Higher probability assigned to the recognized emotion was also associated with lower valence and arousal mismatch. Consequently, an incorrect top-choice label did not necessarily imply that the social signal had been lost. Receivers often retained the general valence, arousal, or relational character of the intended touch while confusing it with a closely related emotion.

These results are broadly comparable with earlier work reporting approximately 45% recognition across six machine-rendered social-touch scenarios (Salvato et al., 2021) and 57% for direct touch between close partners without visual access to the toucher (Hauser et al., 2019). The present study differs substantially in its emotion set, participants, chance level, body regions, and rendering method, so direct performance comparisons should be interpreted cautiously. However, our results demonstrate that social touch can remain communicatively meaningful when it is produced live, rendered through spatially distributed vibrotactile feedback, and situated within a visually embodied interaction. Seeing the other person’s avatar movement, contact timing, and response may complement the tactile cues, consistent with evidence that visual animation can support touch-gesture recognition while tactile feedback establishes and disambiguates contact (Richard et al., 2025). Although touchers were explicitly instructed to avoid deliberately conveying the target emotion through non-touch signals, incidental affective cues may still have been conveyed through facial expressions or body or head movement and, in the mediated conditions, through the corresponding tracked avatar motion. The current study does not isolate the causal contribution of these visual and social cues, but it indicates that mediated touch should be understood as a multimodal interpersonal event. Across both realities, some participants nevertheless wanted the contact to feel more “physically compelling” through pressure, force, or rigid collision and stopping cues. These suggestions reflect the limitations of vibration alone to reproduce resistance and sustained mechanical interaction.

4.4.2 Gesture production transfers better than gesture interpretation

The results reveal an important asymmetry between the production and interpretation stages of social-touch communication. Participants produced strongly differentiated gesture profiles for the seven conveyed emotions. Love and Calming were dominated by strokes and squeezes; Happiness incorporated taps, pats, and squeezes; Anger used a broader combination of taps, pats, pokes, and squeezes; Gratitude combined pats, squeezes, and strokes; and Attention was conveyed primarily through taps and pokes. These patterns are consistent with prior evidence that social-touch messages are constructed from systematic combinations of gesture type, contact area, velocity, repetition, and duration (McIntyre et al., 2022). More importantly, this expressive vocabulary remained comparatively stable across real, VR, and MR. Initiator reality altered the gesture distribution reliably only for Attention, and most gesture durations and gesture-specific dynamics did not differ significantly across realities. The few significant differences, such as more frequent Happiness-related squeezing in VR than RR and more frequent Happiness-related patting in MR than RR, were isolated rather than evidence of a general restructuring of touch behavior. Participants therefore appeared able to translate their intended touch actions into avatar-mediated movement without substantially changing how most emotions were expressed. Participants’ remarks highlight the main MR concerns including hand registration, tracking, and passthrough rendering. Participants occasionally observed misalignment between their physical hand and its virtual representation. This visuomotor discrepancy particularly appeared more disruptive for initiators while conveying gestures to remote avatars, who depended directly on correspondence between their intended movement and the rendered hand, than for receivers. These comments reinforce the importance of accurate spatial registration and stable rendering for MR-mediated touch, even when the overall structure of the gestures being produced remains stable.

The gesture-classification results provide stronger evidence for this transfer. A classifier trained on gesture use and dynamics identified the conveyed emotion with approximately 78% accuracy under both stratified and leave-one-initiator-out validation. When trained in one initiator reality and evaluated in another, accuracy remained between approximately 71% and 79%, far above the seven-class chance level. Anger and Attention were especially separable, whereas most errors involved Love, Sadness, and Calming. The similarity between the classifier’s errors and the receivers’ perceptual confusions is informative: ambiguity among these emotions arose partly because they shared expressive structures, not reality differences.

In contrast, interpretation depended more strongly on the receiver’s reality. Initiator reality produced no significant overall effect on recognition. Receiver reality, however, affected recognition overall, with Happiness and Gratitude recognized more accurately when the receiver was in VR than in MR. The directional Happiness difference between MR–VR and VR–MR further demonstrates that reversing who occupied each reality could change the outcome even though the same two realities were involved. The principal cross-reality asymmetry therefore emerged after the gesture had been produced: the receiver’s perceptual and embodied setting shaped how the rendered action was decoded.

On analyzing gesture dynamics, faster strokes and more frequent squeezes, taps, pats, and pokes were associated with greater receiver arousal, and these relationships were comparatively stable across receiver realities. By contrast, relationships with valence were more dependent on gesture and receiver setting, particularly for stroking, squeezing, and poking. This is consistent with previous findings that velocity and intensity influence both arousal and valence in direct and mediated touch (Zhu et al., 2022; Xu et al., 2021), while extending those relationships to live, reciprocal interaction across multiple realities. A rapid or repeated movement may consistently communicate activation, but whether that activation feels affectionate, playful, unpleasant, or intrusive depends more strongly on the visual, bodily, and social context in which it is received.

4.4.3 VR and MR support different embodied interaction styles

VR and MR did not form a simple hierarchy in which one reality consistently offered a better social-touch experience. They supported different ways of being embodied with and interacting with another person. MR produced higher ownership, multi-sensory embodiment, and total embodiment scores than VR. Its advantage therefore concerned deeper bodily integration rather than how the body looked. Seeing one’s physical body and surroundings while receiving spatially corresponding haptic feedback may have strengthened the continuity between physical and mediated experience. Responses to MR were particularly positive because the remote avatar appeared within the participant’s physical surroundings, and several participants suggested that such interaction could remain “engaging in frequent or everyday use”. Haptic-experience ratings similarly showed higher autotelics in MR, suggesting a more intrinsically rewarding or enjoyable experience. At the same time, realism, harmony, discord, and the overall haptic-experience did not differ between realities. MR’s advantage was therefore not attributable to a general increase in perceived haptic quality. Instead, embedding remote touch within the participant’s visible body and environment appears to have changed the context and personal relevance of the sensation.

VR supported a different set of experiential qualities. It produced higher involvement, co-presence, and behavioral interdependence, indicating stronger absorption in the interaction, a greater feeling of sharing the virtual space, and closer perceived coupling between partners’ actions. MR produced higher attention. Overall social presence and general presence did not differ reliably, which means that the two realities feature different components of the experience rather than different overall amounts of presence. VR created a visually coherent shared world organized around the remote partner, whereas MR placed the remote partner within a richer but less fully shared physical context.

The observed behaviors closely reflected this distinction. Participants remained in VR longer, accumulated more interpersonal touch time, spent a greater proportion of the session touching, and explored a broader set of gestures that lasted longer on average, and collaborative actions. Participants frequently incorporated touch into playful or game-like activities, including synchronized clapping, tag, hand-chasing, treating the virtual environment as a space for exploration, performance, and play. Participants also reported fewer VR-specific difficulties. Their most common requests were “greater control over the virtual background” and a “larger range of objects or activities to experiment with”.

MR interactions were generally more selective and physically grounded. Participants frequently performed hand alignment, handshakes, guided movements, and other coordinated contacts. Greeting rituals and collaborative object activities occurred in both realities, but MR appeared to encourage users to confirm spatial correspondence between the physical hand, the rendered contact, and the remote avatar before expanding the interaction. Stronger embodiment and usability therefore did not necessarily result in more touching.

This grounding also introduced an MR-specific challenge. When a remote avatar appeared within the visible physical environment, participants expected it to interact meaningfully with nearby real objects. Some anticipated that the avatar could move a physical object or that their own contact with such an object would be communicated to the remote partner. MR therefore created a gap between visual co-location and functional co-location: the avatar appeared to inhabit the room without sharing all of its physical affordances. Maintaining credible social interaction in MR may thus require registered virtual counterparts of physical objects, instrumented props, or explicit cues indicating which elements are mutually interactive.

The results also caution that these interaction styles were shaped by prior exposure. Pairs who completed Free Exploration before Prescribed Touch interacted longer, touched more frequently, and explored a broader repertoire of gestures and actions. The VR/MR differences in several exploratory measures were also larger in this group. Completing the structured task first may have provided familiarity, but it may also have introduced fatigue or implicitly taught participants a narrower repertoire centered on the prescribed gestures. Similarly, an overall MR preference was especially strong among participants who completed Free Exploration first, whereas those who completed it later were evenly divided. These findings tell us that open-ended social behavior cannot be treated as independent of participants’ previous tasks and expectations.

Finally, physical comfort remains relevant when selecting interaction styles, especially for prolonged usage and exposure. During prescribed-touch experiment, which was about 2 h long, MR produced greater changes in oculomotor and total simulator-sickness scores than VR, and the reported symptoms were primarily fatigue and eyestrain. Participants’ comments suggest that misalignment between the physical hand and its virtual representation also contributed to these oculomotor symptoms. Integrating passthrough imagery, virtual avatars, physical hand motion, and nearby environmental geometry may impose perceptual demands even while improving embodiment. Thus, MR’s grounding benefits should be considered alongside the cost of maintaining stable visual-physical correspondence over extended interactions.

4.4.4 Design implications

4.4.4.1 Design for communicative touch, not actuator outputs

Our findings suggest that future social touch systems should prioritize preserving the communicative structure of touch rather than reproducing identical haptic signals across devices. Throughout the study, participants conveyed emotions using combinations of gesture type, contact location, duration, velocity, and repetition that remained relatively consistent across realities. At the same time, emotions with similar gesture structure frequently overlapped perceptually, indicating that the social meaning of touch arises from the temporal and spatial organization of the interaction rather than from any single haptic cue. Instead of encoding fixed emotion-to-actuation mappings, systems should preserve the touch action itself and allow receiving devices to render that interaction according to their capabilities. This approach is particularly important for heterogeneous XR systems, where users may interact through devices with different haptic modalities, resolutions, or actuation technologies. Representing touch at the level of communicative gestures rather than actuator commands provides a flexible foundation for future cross-device social touch systems.

4.4.4.2 Design for receiver perception rather than symmetric rendering

Our results further indicate that successful social touch communication depends as much on how touch is perceived as on how it is produced. Gesture production remained comparatively stable across realities, whereas emotion recognition and affective alignment were more strongly influenced by the receiver’s perceptual context. In particular, receiver reality affected recognition more consistently than initiator reality, suggesting that reproducing identical haptic signals on both sides of an interaction does not necessarily produce equivalent social experiences. Future rendering algorithms should therefore optimize for perceptual equivalence rather than signal equivalence. Contact onset, spatial correspondence, intensity scaling, and multimodal cues may need to be adapted to the perceptual characteristics of the receiving user while preserving the underlying communicative intent. Such receiver-aware rendering may become increasingly important as cross-reality communication expands across diverse XR platforms.

4.4.4.3 Co-design embodiment, avatars, and haptics

Social touch cannot be designed as an isolated haptic interaction. Throughout both the prescribed-touch and free-exploration studies, the tactile experience was shaped by the surrounding embodied context, including avatar movement, spatial alignment, visual contact, and users’ sense of embodiment. Differences between VR and MR can be explained by the complete multisensory experience through which touch was interpreted. Future social XR systems should consequently treat avatars, embodiment, visual behavior, and haptic feedback as components of a single communication channel rather than as independent subsystems. Designing these elements jointly may better support the social meaning of touch than improving haptic fidelity alone.

4.4.4.4 Design different realities for different social goals

VR and MR appear to support distinct forms of social interaction. Neither reality was consistently superior across the measures we examined; instead, each emphasized different aspects of embodied communication. Future systems should therefore adapt to the interaction reality according to the intended social context. Applications centered on collaborative play, exploration, or expressive interaction may benefit from the immersive flexibility of VR, whereas everyday communication, emotionally supportive interactions, or situations requiring awareness of the surrounding environment may benefit from the grounded nature of MR. VR and MR are not competing alternatives, so treating them as complementary communication environments may ultimately provide more effective and inclusive social XR experiences.

4.4.5 Limitations and future work

The current system renders social touch using wearable vibrotactile feedback, which captures degree of contact and spatiotemporal motion but cannot reproduce many of the mechanical properties of interpersonal touch, such as sustained pressure, skin deformation, or force exchange. Participant feedback similarly highlighted a desire for richer collision cues and physical reciprocity. Future work should therefore investigate multimodal haptic interfaces that combine vibration with pressure, force, skin stretch, or other tactile modalities that can be used in such a real-time bidirectional communication setting in XR and other digital communication platforms. Moreover, our current design is limited to actuation on the hands and forearms. Expanding to full-body haptic feedback would pose significant challenges in terms of hardware design and power consumption. To address these limitations, future research must also explore alternative actuation techniques capable of replicating complex sensations across the entire body in real-time XR interactions with minimal latency.

Our evaluation was conducted with healthy adult participants in controlled laboratory settings using current-generation VR and MR hardware, where all but one participant pair knew each other prior to the experiment. Social touch is strongly shaped by interpersonal relationships, context, and culture, while limitations in current MR tracking and passthrough quality likely influenced aspects of embodiment and interaction. Future studies should examine more diverse user populations including strangers, long-term use, and emerging XR hardware with improved spatial registration and tracking. Future evaluations of mediated social touch should distinguish between gesture production and gesture interpretation, as communication may change because users produce different touch behaviors, systems render them differently, or receivers interpret identical cues differently. We also observed that prescribed-touch tasks influenced subsequent free exploration, highlighting the importance of carefully considering task order or separating participant populations when studying social touch behavior.

Finally, our XR system is designed to be device-agnostic, enabling researchers to integrate and test their own devices within our framework. As future work, we plan to evaluate our XR system using haptic feedback from commercially available devices (e.g., Ultraleap module or bHaptic gloves and vests), to further strengthen our claim and broaden the applicability of our system. This is a key step toward making our system open source, enabling researchers to develop, evaluate, and compare their own haptic devices within our XR platform. The scope of this XR framework can be expanded, exploring its potential as a benchmark application to evaluate and compare tactile devices (Banerjee et al., 2023) and social touch interaction paradigms.

5 Conclusion

We presented and evaluated a shared XR framework supporting bidirectional avatar-mediated social touch across Virtual Reality (VR), Mixed Reality (MR), and asymmetric cross-reality interaction. Through controlled emotion-communication tasks and open-ended social interaction, we examined how social touch is produced, perceived, and experienced when physically distant users communicate through virtual bodies.

Our findings show that avatar-mediated social touch remains meaningful across heterogeneous realities, although different stages of the communication process transfer differently. Participants preserved a recognizable vocabulary of touch gestures across real, VR, and MR interactions, whereas the interpretation of those gestures depended more strongly on the receiver’s perceptual context than on the initiator’s reality. Rather than representing different levels of the same experience, VR and MR supported complementary forms of embodied interaction, with VR encouraging more exploratory and playful behavior and MR promoting stronger embodiment and more physically grounded interaction. Together, these findings suggest that future social XR systems should focus not only on reproducing physical touch, but on preserving its communicative function across diverse embodied experiences.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Michael Wiesing, University of Barcelona, Spain

Reviewed by: Marco Fyfe Pietro Gillies, Goldsmiths University of London, United Kingdom

Parthan Olikkal, University of Maryland, Baltimore County, United States