Automated detection of mutual gaze in mobile eye-tracking studies of multi-party interactions

Introduction

When we inspect our environment, we continuously move our eyes and direct our gaze towards external objects of interest. Myriad laboratory studies in health and cognitive sciences have used eye-tracking methodology to understand microscopic processes involved in oculomotor control, pathologies of vision, and visual perception and cognition. Most of this research has used stationary, screen-based eye-tracking, recording gaze on fixed visual displays under controlled viewing conditions. Although screen-based eye-tracking constitutes a valid methodology for studying visual information processing, it does not capture how humans allocate gaze in naturally occurring, everyday social interaction. The label “in-the-wild” has been coined as an umbrella term for research conducted without the experimental control typically required in laboratory eye-tracking, pursuing the goal that participants feel less behaviorally restricted and exhibit more natural interaction with their environment (Lappi, 2015) and with each other (Stukenbrock, 2018a).

One field of research that considers data collection in natural, real-world environments paramount has been social interaction research, particularly conversation analysis (Mondada, 2013) and interactional linguistics (Couper-Kuhlen and Selting, 2018). Within this field, major research interest has been dedicated to the organization of social encounters and the role of gaze as a functional resource in face-to-face interaction (cf. Auer, 2018; Brône et al., 2017; Goodwin, 1980a; Kendon, 1967; Kleinke, 1986; Stukenbrock, 2018a, b; Zima and Stukenbrock, 2025). Conversation analytic/interactional linguistics (CA/IL) studies have shown, for instance, that gazing at others is relevant for floor management (Auer, 2018; Ho et al., 2015; Rossano, 2012; Weiß, 2020; see Degutyte and Astell, 2021 for review).

Mobile eye-tracking technology has allowed examinations of gaze allocation from participants' first-person perspective in high temporal and spatial resolution. To investigate the diversity of social gaze phenomena, dual mobile eye-tracking designs (Stukenbrock and Zima, 2025) have been employed to analyze, for instance, mutual gaze/eye contact, joint attention, unidirectional eye gaze, or gaze aversion, as resources in in-the-wild interactions between dyads or multi-party configurations.

Since in-the-wild eye-tracking studies typically allow participants to move freely and require multiple units of mobile, head-free recording hardware, they entail methodological challenges and constraints (Chamberlain et al., 2012; De Beugher et al., 2014; Lappi, 2015; Stukenbrock, 2018a, b, 2020, 2023). Stukenbrock (2018a, p. 268f.) notes that deteriorating calibration, data loss, variable lighting conditions, and other environmental factors must be addressed, alongside the need for an external full-shot camera perspective, discrepancies between participants' changing real-world spatial configurations and their (non-iconic) representation in synchronized data. Similarly, various research groups highlight operational challenges arising from multiple recording streams, which increase analytical complexity (e.g., precise temporal synchronization), labor demands (e.g., computing power, data collection and processing), and financial costs (Bauer et al., 2026; Valtakari et al., 2021). One of the main difficulties, however, lies in spatial synchronization of multiple, simultaneous eye-tracking recordings.

Mobile eye-tracking systems typically estimate gaze positions as image coordinates of a built-in camera that records participants' field of view (head unit coordinate system). This renders the gaze data's interpretability dependent on the wearer's head orientation relative to the looked-at object. To demonstrate why this is problematic, imagine a participant moving through a museum and looking at an exhibit (Stukenbrock, 2020; 2023). As the participant moves towards it, the videographic representation of the exhibit dynamically changes size and position in the camera image, continuously moving and resizing a meaningful area-of-interest (AOI). Although the participant may be fixating the same part of the exhibit, the positional gaze data, quantified as two-dimensional pixel values, covaries with the perspective that the moving participant takes on the object-of-interest due to the positioning or orienting of his body (head tilt, yaw, or roll, standing closer or further away). This makes it difficult to define stable AOIs and, consequently, to derive analytically meaningful variables from the raw gaze data. In the participant's subjective experience, however, gazing at the same exhibit from different perspectives does not preclude them from apprehending it as a coherent percept and unique cultural phenomenon. This disconnection between perceptual continuity and coordinate instability illustrates the main challenge of contemporary research on in-the-wild eye-tracking data: head-free gaze behavior cannot be meaningfully quantified and interpreted without additional spatial normalization or scene-anchoring procedures or by resorting to tedious manual AOI annotation (Lappi, 2015; Valtakari et al., 2021). To this end, and despite noteworthy advances in the field (e.g., Tobii, 2022, with Assisted Mapping; Hooge et al., 2026), Lappi's (2015) remark that “no off the shelf solutions exist” (p. 4) unfortunately still holds true. This spatial synchronization challenge becomes even more pronounced in multi-participant mobile eye-tracking, where the object of interest is as variable as another human face seen in natural interaction.

Starting in the second half of the 20th century, researchers in the social sciences began to examine mutual gaze as a particularly salient phenomenon related to the fundamental interactional dynamics of intimacy and aggression in human sociality (Rossano, 2012). Referring to “the almost mysterious quality of the mutual gaze”, Kendon (1967, p. 22) observed that “one enters with one another into the most pure and intimate kind of direct relationship”. Accordingly, empirical interest was sparked to examine the duration of mutual gaze, how it is established and resolved, as well as its functions in the organization of turn-taking and participation. Early studies measured mutual gaze by relying largely on direct human coding (Kendon, 1967; for review, see Kleinke, 1986), either through live observation or video analysis. More recent work, in contrast, has increasingly employed eye-tracking technologies, enabling less obtrusive, more objective, and higher-fidelity measurements (Zima et al., 2025). In a recent review, Jongerius et al. (2020) compared quantitative approaches to measuring mutual gaze. To our knowledge, no software currently provides reliable, user-friendly face detection and recognition tools that automatically generate AOIs for the eye region and classify mobile eye-tracking data accordingly. As a result, even recent studies (Auer, 2018; Weiß, 2020; Zima and Stukenbrock, 2025) have relied on manual annotation of mutual gaze events by reviewing gaze cursor movements and locations in synchronized eye-tracking videos. To move beyond such labor-intensive, potentially error-prone manual coding procedures, computer-vision applications for detecting bodies, heads, or faces in video images have gained popularity among technically proficient eye-tracking researchers. Some of these approaches apply a strict criterion, counting only fixations on the eye region (Hessels et al., 2018). Others, by contrast, classify looks to the entire face (Jongerius et al., 2021) or any gaze directed at the co-participant as eye gaze (De Beugher et al., 2014). Consequently, substantial variation exists in how gaze samples are classified as presumably directed at another's eyes. The absence of standardized measurement and analytic procedures has limited comparability and replicability across studies, thereby hampering progress in understanding mutual gaze and its role in social interaction.

Against the background of the current state of the art, the goal of this study is to develop an analytic procedure that uses computer-vision programs for fiducial marker tracking, face detection, and face identification to automate AOI placement on co-participants' faces and enable efficient analysis of social gaze. We introduce a pipeline that extracts unilateral gaze and mutual gaze and apply it to conversations between four-person multi-party configurations (n = 20). Subsequently, we examine the resulting gaze metrics in exploratory statistical analyses and discuss them in relation to previous research and theory, as well as the pipeline's reliability and validity.

Literature review

Before presenting the pipeline, we briefly review a selection of computer-vision–based approaches that could be applied to the automated detection of mutual gaze from mobile eye-tracking data in studies of social interaction (De Beugher et al., 2014; Jongerius et al., 2021; Matar et al., 2025; Tschacher et al., 2021).

Major methodological advances were achieved by a research team including Brône, Oben, Goedemé, and collaborators across a series of studies (Brône et al., 2011; Callemein et al., 2019; De Beugher et al., 2014). They implemented and compared the performance of object recognition and detection algorithms (e.g., OpenPose) to define AOIs on human bodies, limbs, and heads in mobile eye-tracking recordings. Most notably, they avoided marker-based detection to move beyond laboratory constraints and enable data collection in settings where participants must not sit still. Their research established the technical feasibility of applying deep learning computer-vision methods to mobile eye-tracking data and inspired subsequent developments. One such development is a software application worked out and validated by Jongerius et al. (2021). It provides a graphical user interface with clickable buttons for the head detection procedures (based on OpenPose), manual AOI editing, and a semi-automated person identification feature (Person Re-ID). The latter enables users to click on detected faces in the video stream and identify them manually, feeding human-validated recognition data into the program for consistent automatized identification.¹

These advancements represent clear progress and substantially reduce manual coding time, and the algorithms achieve performance levels that approximate human annotation. However, the required manual intervention for adjusting and fine-tuning AOI parameters may remain labor-intensive. Furthermore, although Jongerius et al. (2021) report robustness of the “Person Re-ID” identification technique for a small multi-party subsample, it is uncertain how well it works in multi-party configurations in in-the-wild recordings, which likely involve luminance variation, lower image quality, participant movement/motion blur, only partial visibility of torsos, or similar-looking participants viewed from various angles and with partially occluded faces (e.g., by eye-tracking glasses or superimposed gaze cursors). The stability of “Person Re-ID” under such conditions remains to be verified. Nonetheless, their work demonstrated that automated AOI detection based on contemporary face detection algorithms can achieve accuracy levels comparable to human coders for mobile eye-tracking systems, considerably reducing the time and labor required. The marker-free design, integrated identification mechanisms, and availability of a graphical user interface enhance usability, particularly for researchers without extensive programming experience. Importantly, these validation studies deliver a crucial methodological premise for the present approach, as they establish the viability of face- and object-detection programs for the analysis of eye-tracking video data.

While the technological advances are substantial, the measurement of a single participant's gaze at a co-participant contributes little to our understanding of dyadic and multi-party interaction. In interaction research, a central question is not only how one person allocates gaze, but how the gaze behavior of two or more interactants emerges in and relates to processes of mutual monitoring and interpersonal coordination. The study of social gaze therefore requires temporally aligned data streams from two or more individuals, yet none of the cited approaches addresses the synchronization challenges inherent in multi-eye-tracking setups (Valtakari et al., 2021). Nor do they propose analytical frameworks for operationalizing social gaze dynamics, such as how to detect mutual gaze in two data streams. Although these aspects were not the primary focus of the respective studies, their absence constrains the applicability of the methods in dyadic and multi-party interaction research.

Beyond these technical limitations, an inferential issue remains. The resulting data typically consist of temporal markers and duration measures (of, e.g., fixations on a co-participant's face). Conventional summary statistics derived from these measures, such as mean durations or total proportions (cf. Matar et al., 2025), provide only limited insight into the interactional significance of mutual gaze. Consequently, while the quantitative outputs are precise, they do not in themselves capture the qualitative impact of social gaze and how it interfaces with other interactional modalities, such as talk (Auer, 2021a, b; Brône and Oben, 2018; Holler and Kendrick, 2015; Zima and Stukenbrock, 2025) and gesture (Kendrick et al., 2023; Streeck, 1993, 2002; Stukenbrock, 2014, 2018a, b, 2020; cf. also Jongerius et al., 2020).

Against this background, Tschacher et al. (2021) were among the first to examine social gaze in dyadic interaction using a multimodal mixed methods design that built on prior qualitative work (Stukenbrock, 2020). Using scene video recordings from TobiiPro Glasses 2, they computed synchrony measures for head movement, eye movement, and mutual gaze in a single-case dyad, conceptualizing synchrony as an index of temporally coupled eye movement dynamics across the course of a conversation. Methodologically, eye movement was operationalized as changes in the Euclidean distance between the scene video frame's origin and the gaze cursor centroid, and mutual gaze as the near-zero approximation of both participants’ Euclidean distance between the gaze cursor centroid and the interlocutor's facial region. Synchrony was quantified using surrogate synchrony (SUSY; Tschacher and Meier, 2020) via windowed cross-correlations across ±3-second lags in 30-second segments, with effect sizes (ES_noAbs) calculated as the difference between observed and surrogate (shuffled) segment correlations divided by the surrogate standard deviation, yielding a standardized measure of above-chance temporal alignment. Positive ES_noAbs values indicate in-phase synchrony, whereas negative values reflect anti-phase dynamics suggestive of leader-follower patterns. The authors reported anti-phase synchrony for eye movements, linking increasing movement in one interlocutor to decreasing movement in the other, which could be interpreted in relation to qualitative interactional phenomena such as floor management in conversation: attentive listeners maintain gaze on the speaker (Goodwin, 1980a) while speakers gaze away more often (Goodwin, 1980a; Kendon, 1967) and shift gaze back to the interlocutors, e.g., for checking (Kendon, 1967) or monitoring (Goodwin, 1980b; Stukenbrock, 2020), for feedback elicitation (Zima, 2020), response mobilization (Stivers and Rossano, 2010), turn allocation (Weiß, 2020), addressee selection (Auer, 2021a), as well as co-addressing multiple participants (Auer, 2021b). Mutual gaze synchrony analyses (i.e., temporal coupling of eye movements towards each other's faces) yielded compatible results, in that one's gaze shifts towards the other's face were reciprocated with mutual gaze in anti-phasic fashion (Tschacher et al., 2021). However, substantial missing data—due to reliance on computer-vision-derived gaze cursor coordinates rather than raw gaze data and inconsistent face detection—limited interpretability. Nonetheless, the study demonstrated proof-of-concept that surrogate synchrony (SUSY) can generate quantitative indices of temporal coupling across multiple behavioral modalities, including eye movements and mutual gaze, thereby constituting a methodological innovation and providing practical recommendations (e.g., the use of ArUco markers) that informed subsequent multi-party implementations, including the present work.

While writing this manuscript, we became aware of a relevant preprint by Matar et al. (2025), who explicitly built on the synchrony framework proposed by Tschacher et al. (2021) and extended it to a single-case triadic interaction. Adopting the surrogate synchrony (SUSY) approach and its multivariate complement (mv-SUSY; Meier and Tschacher, 2021), they computed gaze synchrony between three participants, operationalizing joint attention as any frame in which two participants simultaneously gazed at the third participant's face, while mutual gaze was defined as reciprocated face gaze between two interactants within the same frame. They elaborated their setup by positioning a single ArUco marker centrally between three participants seated in a triangular arrangement at a round table, so that each participant viewed the marker centrally in their visual field. To make gaze movements comparable, they relied on ArUco marker detection (Garrido-Jurado et al., 2014) and computed horizontal and vertical distances of participants' gaze coordinates to the ArUco marker centroid. For face detection, they applied TobiiPro's proprietary Assisted Mapping (cf. Tobii, 2022, p. 85) to scene video snapshot images of co-participants to place facial AOIs. The software then binary-coded each gaze sample as either within or outside the manually labeled facial AOIs. Using SUSY, dyadic mutual gaze synchrony was computed from these binary AOI-hit time series, resulting in significant mutual gaze synchrony. Triadic face gaze synchrony as computed by mv-SUSY, attested significant presence of face gaze synchrony between all three participants, i.e., all three dyads reciprocated mutual gaze with each other in a coordinated fashion (Tschacher et al., 2021). These results were interpreted as indicative of synchronized social attention, showing that listeners tended to look at the speaker while speakers reciprocated gaze at the others above chance levels. In this sense, the study supported the feasibility of extending surrogate synchrony methods (Meier and Tschacher, 2021; Tschacher and Meier, 2020) beyond dyads and demonstrated that even a single ArUco marker facilitates the organization of raw gaze data in triadic eye-tracking setups. However, Matar et al. (2025) pipeline relies on proprietary functions for the crucial steps of face detection and gaze mapping, limiting transparency and reproducibility for researchers without access to TobiiPro's software. Moreover, participants were instructed to maintain relatively stable and upright postures, and effectivity of the pipeline depends critically on the visibility of the centrally placed marker; spontaneous movement (e.g., head turns, posture or orientation changes, or expressive gestures) would likely disrupt marker visibility and compromise analysis. This raises concerns about scalability to more naturalistic interactions as well as larger groups, in which such movement forms an integral part. In contrast, the pipeline proposed in the present paper permits greater freedom of movement while still reliably detecting and identifying co-participants, and implementing entirely open-source software solutions.

Methodology

The analysis pipeline was developed for mutual gaze detection in multi-party configurations of four or more participants.² We applied the pipeline to eye-tracking data collected in the context of a larger project (see Dirker et al., in press). Participants (n = 100) were invited as groups of four (henceforth: quartets; n = 25) with most participants familiar with their co-participants. Wearing eye-tracking glasses (2 × TobiiPro Glasses 3, 2 × TobiiPro Glasses 2, sampling pupil data at 50 Hz), quartets were seated at a table and assigned to plan a 5-course meal in a spontaneous conversation (avg. duration = 14m07s). At the beginning and end of the conversation, participants looked at a clapperboard to obtain a visible signal in the scene video, which was used for temporal synchronization of the eye-tracking recordings (cf. Valtakari et al., 2021; Ho et al., 2015; Macdonald and Tatler, 2018; Stukenbrock, 2018a, b). Figure 1 depicts a schematic of the pipeline.

Step 1. Scene video export

TobiiPro eye-tracking glasses have scene cameras built into the nose bridge that record video images at 25 fps in Full HD resolution (1,920 × 1,080px²) of the participants' field of view, covering 95 visual degrees horizontally and 63 degrees vertically (TobiiPro, 2021). The gaze data are recorded separately by eye-tracking cameras that are integrated into the lenses and the wearer's gaze location is quantified as scene video image coordinates. These gaze coordinates can be mapped onto the scene video as a circular gaze cursor (see Figure 2) for observing dynamic gaze allocation as it emerges throughout the interaction. Scene video recordings were exported from the software TobiiPro Lab. These video files are important raw data and input to ArUco (see Step 1.1) and face detection procedures (see Step 1.2).

Step 1.1. ArUco marker detection

The first procedure is to scan the scene videos for fiducial markers. Participants were recorded in rooms decorated with fiducial markers of the ArUco system (Garrido-Jurado et al., 2014), which were attached to flat vertical surfaces of walls and furniture. ArUco markers look like low-resolution QR codes and are essentially square grids made up of black and white cells in different bit grid sizes.³ We used 24 unique markers with a 6 × 6 grid (ID range: 11–34), colored in the white cells⁴, and printed them on A4 paper. Marker positioning ensured that every participant saw at least two markers whenever they were facing another participant in the room (see Figure 5 in the Supplementary Materials for marker placement). No markers were attached to the floor or the ceiling, although detection from horizontal surfaces is equally possible.

The ArUco detection script (aruco.py; cf. Supplementary Materials) was run in a virtual environment with Python 3.13, relying mainly on functions of the OpenCV library (Bradski, 2000). It was set up to store all identified markers' ID number, their two-dimensional (x/y) corner positions, as well as their centroid for each frame. As output, the script generates a comma-separated file for each processed scene video with continuous frame numbers (25 fps) providing temporal order and synchronizability. Figure 2 below visualizes the result of the ArUco detection script on a scene video screenshot.

Step 1.2. Face detection

ArUco detection by itself is insufficient to determine the exact locations of co-participants' faces in the video images. Participants' head movements (roll, pitch, yaw) or seating postural changes cause variation in relative spatial relations (angles, distances) between ArUco markers and their co-participants. Under such conditions, relying solely on predefined image regions (as presumed face locations) anchored to static, trackable objects (Tschacher et al., 2021), like ArUco markers, is insufficient. Therefore, to efficiently locate the image areas where faces were visible, a face detection model was used (cf. Callemein et al., 2019; Jongerius et al., 2021).

Face detection (facedet.py; cf. Supplementary Materials) was based on RetinaFace (v.1.1.1; Deng et al., 2019) and executed in a virtual environment running Python 3.9. RetinaFace is a deep learning-based model that reliably detects faces in images with even suboptimal quality, be it in terms of brightness or acuity, when faces are partially occluded or visible only from unconventional angles. For each detected face per frame, the RetinaFace model returns image coordinates of facial landmarks (both eyes, mouth corners, and the nose tip) as well as the bounding boxes (i.e., the rectangular areas enclosing the visible face from chin to forehead and ear to ear) at high precision (Deng et al., 2019: Figure 6⁵). These data were stored in the output files [e.g., (participant id)_faces.csv]. The bounding box coordinates are later used to define facial AOIs for social gaze event detection (see Step 2.1).

Since both the ArUco and face location data sets share a running sequence of frame numbers (based on the unique scene video file), they are now merged by frame number. A visualization of this combined data is shown in Figure 2.

Step 1.3. Co-participant face identification

While the face detection and localization procedures work reliably, they provide no mechanism for consistent face identification across consecutive video frames. As a result, the same participant may be labeled “face_1” in one frame and “face_2” in another. Because the pipeline aims to identify mutual gaze segments between specific individuals in multi-participant configurations, it must supplement detection with a procedure for stable face identification. We therefore distinguish between automated face detection (Step 1.2), i.e., the frame-wise localization of faces in images, and automated face identification, i.e., assignment of identity to each co-participant. This section describes the identification procedure, which is essential for determining who is looking at whom in multi-person interaction.

Since participants remained seated, their co-participants' faces appeared in roughly predictable positions with respect to the ArUco markers. Consider again Figure 2, where the participant was looking toward the person seated across, and faces of other co-participants are visible in their field of view, as well as a subset of ArUco markers 22, 34, 19, 28, 25, and 16. Figure 3 below depicts this data in a simplified schematic. The face identification script developed for this project (face_id.R; cf. Supplementary Materials) runs through multiple processing steps that mainly rely on the relative locations of faces to their surrounding ArUco markers.

Face identification begins by setting basic conditional statements. Based on the centroid x-coordinates of the ArUco markers and faces, conditions are set that if a given facial centroid x-coordinate is, e.g., larger than the centroid x-coordinate of marker 34 but smaller than the centroid x-coordinate of marker 25, then the face detected within the area confined to this x-axis region is “personacross” in our setup. On the other hand, if the centroid x-coordinate of a face is smaller than the centroid x-coordinate of marker 34, but larger than the centroid x-coordinate of marker 22 minus 200 pixels, then that face is “personleft”. When a detected face's x-coordinates are larger than marker 25 and smaller than marker 16 plus 250 pixels, that face is “personright”.

To aid the researcher in setting up these distance-based conditional statements, the script is programmed to output random samples of 100 schematic frame images (similar to Figure 3) in which at least two faces and two markers were detected. It is recommended that the researcher view these images to fine-tune the conditional statements for each of the detected faces and markers. Because participants moved their heads and torsos, altering the spatial relations between markers and faces across the video, conditional statements must be checked and adjusted across multiple sampled frames to ensure they identify reliably. Note that this step of face identification is time-consuming and requires human control and parameter fine-tuning to achieve valid results.

Despite fine-tuned conditioning, some faces may remain unidentifiable or improperly labeled due to missing markers or noise. Two issues arise.

First, only one out of two or three detected faces may be identified through markers. In this case, the identified face may be used to determine the other(s). Consider again Figure 3. If “personacross” was identified, further conditioning may allow identification of “personleft” (face centroid x-coordinate is smaller than that of “personacross”) and “personright” (larger x-coordinate of face centroid). If either “personleft” or “personright” were identified, then their facial centroid x-distance to “personacross” is smaller than to the respective other, allowing further identification through simple conditioning (e.g., distance to “personacross” is no larger than, e.g., 1,400 pixels). We term this step prior-identification assignment.

Second, in frames where only one ArUco marker and one face were visible, the conditioning above likely fails to assign a unique face label, as identification was set up to rely on multiple markers for validity. Therefore, in a subsequent step, termed distance-threshold assignment, means and standard deviations of the horizontal distances between face and marker centers were computed for all identified face samples. These values were then used to evaluate whether the distances of yet unidentified faces to a given marker fell within defined ranges (e.g., mean ± 2SD). This procedure was applied stepwise, first with a strict ±2 SD threshold and then with a more liberal ±4 SD threshold. The broader margins allowed labeling of outlier faces in frames where participants may have rolled or yawed their heads to unusual poses.

The labeled and unlabeled face data are then joined and ordered by frame number. On this data, further labeling is performed using nearest-neighbor logic. For each frame with an unlabeled face, a window of adjacent ±10 frames (400 ms) is searched. If at least one identified face is found within this window, Euclidean distance between the unlabeled face centroid and the labeled face(s) is computed. The unlabeled face is then identified based on the spatially closest labeled face (i.e., the one with the minimum Euclidean distance), under the premise that within ±400 ms, head movement will not shift the face far enough to invalidate nearest-neighbor matching. If no labeled faces are present within the window, the face remains unlabeled. This procedure effectively pastes known labels to nearby frames, filling gaps in the annotation while preserving spatial consistency.

In previous steps, distinct face samples may have been assigned identical labels, resulting in duplicates. Consequently, whenever two or more face samples within a single frame shared the same co-participant label, the labels were removed, and the face samples underwent the following procedure.

In frames where two unidentified samples remained, faces were identified using a combination of facial bounding box circumradius statistics and centroid x-coordinate. In our quartet setup, the participant seated across from the glasses-wearer appeared farthest away and exhibited the smallest face radius, identifying them as “personacross”. The remaining co-participants, seated more closely and producing larger radii, were then differentiated as “personleft” and “personright” based on their centroid x-coordinates relative to “personacross” or each other (like prior-identification assignment).

In most remaining unidentified samples, only one face but no ArUco markers were visible. Predicting isolated face identity is tricky, but at this stage in the pipeline, enough identified samples allow for statistical inference. Samples identified as “personacross”, for example, allow for a precise calculation of central tendency of face circumradius distribution. In a first step, then, all isolated face samples with a radius smaller than +1SD above the mean radius of “personacross” were identified as such. Isolated face samples with radii larger than that were most likely “personleft” or “personright”. In a second step, these were distinguished based on the horizontal distance of the nose tip to the facial bounding box centroid. As a rule of thumb, from the glasses wearer's perspective, the nose tip of “personleft” should generally be right of their facial centroid (ergo, horizontal distance is positive) and left for “personright” (horizontal distance is negative).

The dataset resulting from Step 1.3 contains the scene video coordinates of co-participants' faces with identity labels. Before merging the face with the gaze data, a last human validity check is conducted, in which a random selection of frames (default: 1% of all frames; avg. 215 in our sample) are output as schematic images (cf. Figure 3) and can be reviewed.

Step 1.4. Gaze data processing

TobiiPro Glasses have inbuilt infrared eye cameras that record images of the wearer's eyes, and the pupil is calibrated to scene video coordinates using the pupil-and-corneal-reflection method. Recordings were imported into TobiiPro Lab (v. 1.217), the manufacturer's data processing suite. Raw data (averaged across both eyes) was exported as participant-specific data tables comprising gaze coordinates and pupil sizes (at 50 Hz) ordered by continuous timestamps in microseconds. Linear interpolation was applied to the raw gaze data, filling sampling gaps of up to 1,000 milliseconds.

Step 2. Synchronizing the data streams

Besides the gaze data, we avail of the identified faces' screen coordinates ordered by numbered sequences of scene video frames. For synchronization purposes, we shut a clapperboard at the beginning and end of each quartet's conversation. We manually scanned the full scene videos for the timestamps in which the clapperboard shut. This timestamp was converted into a frame number relative to scene video recording onset (timestamp(inmilliseconds)1000×25fps), and stored as interaction on- and offsets. With these frame numbers, we filtered the face AOI data for the frames of the quartet conversation.

For the raw gaze data, temporal order was not in video frames but in milliseconds, beginning at 0 and proceeding in 20 ms increments with every sample. TobiiPro's manuals for Glasses 2 and 3 report a frame rate of 25 fps for scene video recording, yielding face AOI data at 25 Hz. The gaze data (50 Hz) was downsampled to match this sampling rate by removing every second observation. Gaze data timestamps were then normalized by subtracting the minimum recorded timestamp from each observation. Then, row-wise differences between consecutive samples were computed. Based on these inter-sample intervals, frame increments were assigned using time windows (0–25 ms = 0, 26–44 ms = 1, 45–74 ms = 2, 75–114 ms = 3) that were derived from analysis of our sample's (n = 100) summary statistics, thereby accounting for short-term data loss or irregular sampling intervals. Cumulative summation of these increments yielded a monotonically increasing frame index aligned with the temporal structure of the scene video. Rather than simply dividing the raw eye-tracking timestamp by 40 ms, this procedure accounted for variance in inter-sample intervals and mapped gaze data onto a 25 fps frame structure (40 ms per frame), ensuring synchronizability of gaze and scene video-derived data.

The gaze data was then filtered according to the critical interaction on- and offsets and joined with the face AOI data by frame number. To facilitate later synchronization with the corresponding data streams of co-participants, the frame count was re-referenced to zero by subtracting its minimum value from all observations.

Step 2.1. Extract unilateral gaze

At this stage, we can now determine whether a participant looked at a co-participant's face. To do so, the procedure is to determine for every gaze sample whether it was located within any of the facial AOIs in each frame.

Facial AOIs were defined as circles. The radius of each circle corresponded to the circumradius of the facial bounding box, that is, the Euclidean distance between the centroid of the facial rectangle and its corner points (see Step 1.2). To allow for variation in spatial accuracy of gaze estimation and still detect slightly-off fixations as a potential face gaze event, an error margin of 0.5 × radius was added (see Figure 2). Any gaze sample measured within this broader facial AOI was labeled as unilateral gaze sample toward a specific co-participant [labeled as: G_at_(participant id)]. To remedy short intervals of data loss, the respective unilateral gaze vectors are interpolated, replacing missing samples with the surrounding values before and after a gap of maximum 200 ms, but only when the participant's gaze was on the same co-participant immediately before and after the brief data-loss interval. Thus, the interpolation is performed under the assumption that such short interruptions are not gaze aversions but rather short-term data loss. This participant-specific data is then stored as comma-separated files “face_ids_plus_gaze_(participant id).csv”.

Step 2.2. Extract mutual gaze

For the computation of mutual gaze, all previous steps must be repeated for all co-participants. The data of the four participants is then synchronized based on frame number (see mg_elan_script.R, Supplementary Materials). For each sample, it was determined whether a unilateral gaze sample of, e.g., participant 01 directed at participant 02 was reciprocated in the frame-equivalent sample of participant 02, that is, whether participant 02 was classified as gazing at participant 01 simultaneously. Such reciprocal classifications within the same video frame constitute mutual gaze (for a similar procedure, see Rühlemann and Barthel, 2025). This was operationalized by joining gaze samples on participant and frame identifiers, with the additional condition that the gazer of one sample corresponded to the gaze addressee of the other, thereby isolating samples where pairs of participants engaged in mutual gaze [labeled as: mg_with_(participant id)].

Step 2.3. Generate social gaze annotations for ELAN

To merge consecutive frame sequences of consistent unilateral or mutual gaze into social gaze events, all frames in which a participant's gaze was directed at another identified face were selected. For each participant pair, selected frames were grouped into coherent stretches whenever the temporal gap between consecutive frames did not exceed a predefined threshold (<400 ms). For each stretch, onset and offset times were recorded, and duration was calculated. This yielded continuous gaze episodes, defined by their start, end, and total length in milliseconds. The timestamps (onsets, offsets) and durations of these stretches of gaze events are translated into a ELAN-compatible format (hh:mm:ss:mss) and labeled with a Tier (here: participant ID), as well as an annotation label indicating the type of social gaze event [i.e., G_at_(participant id) or mg_with_(participant id)], and exported as a comma-separated text file. See Figure 4 below for a screenshot of ELAN (Wittenburg et al., 2006; v. 7.1) with gaze event annotations.

Analysis

To test the usability of the pipeline's output, an exploratory statistical analysis was conducted. To explore different relationships, gaze behavior throughout the quartet interaction was operationalized as subject-level proportions of cumulative unilateral gaze durations (individual face gazing; Analysis 1), co-participant-specific face gaze proportion (Analysis 2), and by-dyad proportions of cumulative mutual gaze durations (Analysis 3). Analyses were conducted at either the individual or dyad level depending on whether the outcome reflected a participant's own gaze behavior (analyses 1–2) or the jointly produced behavior of a participant pair (analysis 3). Generalized linear mixed models (henceforth GLMM) were fitted to these proportion outcomes, with predictors including the seating position of the co-participant, speaking-duration proportions (of the gazing subject in analysis 1; of the gazed-at in analysis 2), age, gender configuration, individual- and pair-level familiarity, movement synchrony measures, and whether the nested dyad had previously completed a separate dyadic task before joining the quartet (for details on the data collection procedure, see Dirker et al., in press). The following variables were extracted or derived as predictors for the models.

The first predictor, the eye-tracking glasses' tracking ratio, was computed for each participant as the proportion of valid gaze samples out of all samples recorded during the quartet interaction. To ensure comparability in data quality, gaze data from 20 quartets were screened for unusually low tracking ratios. Participants in the lowest 10% of tracking ratios were excluded, and the quartets in which they were nested were removed entirely, resulting in a final dataset of 48 participants (12 quartets; 72 nested dyads) with an average tracking ratio of 87 ± 9%. Because lower tracking ratios can artificially reduce observed gaze proportions, tracking ratio was included as a control variable in all statistical models to account for potential confounding by data-quality differences.

Proportions of individual speaking durations were extracted from the verbal transcripts created in ELAN. Following GAT2 conventions (Selting et al., 2011), the transcripts include annotations of paraverbal vocalizations (e.g., sighing, laughing) for each participant. For each subject, the total speaking duration was divided by the overall duration of the quartet interaction to obtain speaking-duration proportions.

Relative seating position was coded as a categorical predictor, distinguishing whether co-participants were seated frontally across from the gazing participant (reference level) or laterally to the right and left. Previous dyad status was included as a logical factor in the dyad-level analyses (2 and 3 below), indicating whether a given nested dyad had completed a separate conversation before reconvening for the quartet interaction.

Familiarity with the co-participant (5-point Likert rating), age, and gender identity were obtained from questionnaires completed after the quartet conversation. Familiarity and age (z-scaled) were included as numerical predictors. In the dyad-level analyses, gender configuration was operationalized as a categorical predictor indicating whether a dyad consisted of mixed- or same-gender participants. For the analysis of mutual gaze, familiarity and tracking ratio were averaged per dyad.

As described in Dirker et al. (in press), movement synchrony measures were derived using a surrogate synchrony method based on windowed cross-correlations (SUSY; Tschacher and Meier, 2020). These measures quantify how closely aligned in time participants' spontaneous body movements (e.g., head turns, gestures, postural adjustments) were during the interaction. In the first analysis, we included individual synchrony contribution, defined as the average of a single participant's dyadic movement synchrony with the three other quartet members (Dirker et al., in press; Gordon et al., 2021). This value reflects how much that participant contributed to overall group synchrony. In the second and third analyses, we used dyadic movement synchrony (SUSY's ES_noAbs), which captures how synchronized each nested dyad's⁶ body movements were across the entire quartet conversation.

Statistical analyses were performed using RStudio (Posit Team, 2025) and the packages glmmTMB (Brooks et al., 2017; version 1.1.14), DHARMa (Hartig, 2024; version 0.4.7), performance (Lüdecke et al., 2021; version 0.15.2), emmeans (Lenth and Piaskowski, 2026; version 2.0.3), and ggplot2 (Wickham, 2016; version 4.0.1). For all three proportion outcomes, a generalized linear mixed-effects model assuming a beta distribution with a logit link was fitted. This approach is appropriate for non-normally distributed outcomes bounded between 0 and 1 and allows the inclusion of random effects structures that account for the hierarchical dependencies in the data.

Results

Exploratory analysis 1: individual face gazing

The first model examined individual differences in participants' (n = 48) face gaze behavior. The dependent variable was each participant's overall proportion of time spent looking at co-participants' faces (and not at the wall, ceiling, or other non-faces) during the quartet interaction. In this analysis, the three co-participants are not distinguished; the goal is to identify predictors associated with an individual's tendency to engage in face gazing. GLMM fixed effects included age (z-scaled), tracking ratio, speaking-duration proportion, and synchrony contribution (z-scaled). To account for the nesting of participants within their specific quartets, the model included a random intercept.

A model-based marginal mean estimate indicated that participants spent approximately 40% of the interaction looking at their co-participants' faces [M^ = .40, 95% CI (.38, .42)]. Modeling results point to two significant predictors (see Table 1 below). As expected, tracking ratio was associated with higher face gazing proportions (β = 3.49, z = 6.2, p < .001), reflecting a mathematical dependency between data availability and the proportion of gaze samples classified as face-directed. Speaking duration proportion was negatively associated with face gazing proportions (β = −1.05, z = −2.6, p < .01). This suggests that the more someone spoke, the less they looked at others' faces. The model provided no evidence that age or individual synchrony contribution predicted overall face-gazing proportions.

Table 1. Summary of parameter estimates and variance components from the GLM model of individual face gaze proportions.

Since the model used a logit link, coefficient estimates are expressed on the log-odds scale and should be interpreted in terms of direction and significance rather than magnitude.

Bold values indicate significance at α < .05.

After accounting for the fixed effects, there was no detectable variance stemming from grouping levels, but due to the hierarchical nature of the data (participants nested in quartets), the random structure was retained. Model fit was evaluated and indicated no evidence of over-dispersion (dispersion = .98, p = .94). Inspecting both the residual distributions and the residual–fitted plots suggested no indication of uneven spread across predicted values.

Exploratory analysis 2: co-participant-specific face gaze

The second model explored predictors of unilateral gaze toward distinct co-participants in the group, with each participant interacting with three others (n = 144). The dependent variable was the co-participant-specific face gaze proportion, that is, how much of a participant's unilateral gaze was directed at each individual co-participant. If gaze were perfectly balanced across the three co-participants, each co-participant-specific proportion would amount to 1/3 of the participant's total face gazing time. Given this hierarchical structure, a GLMM was specified with a random intercept capturing variation among participants nested within groups, and several co-participant-specific variables as predictors. Model syntax followed theoretical relevance and sequential model building guided by AIC, which led to exclusion of age difference (i.e., a dyad-level covariate for age). Several significant predictors emerged (see Table 2).

Table 2. Summary of parameter estimates and variance components from the GLM model of addressee-specific face gaze proportions.

Since the model used a logit link, coefficient estimates are expressed on the log-odds scale and should be interpreted in terms of direction and significance rather than magnitude.

Bold values indicate significance at α < .05.

Consistent with analysis 1, tracking ratio was significantly associated with co-participant-specific face gaze proportions (β = 1.8, z = 4.38, p < .001). A model-based marginal mean estimate indicated that participants spent approximately 13% of the interaction looking at each of their co-participants' faces [M^ = .13, 95% CI (.11, .14)]. Seating position significantly predicted co-participant-specific gaze proportions: for laterally seated co-participants, face gaze proportions were lower than for frontally-across-seated co-participants (β = −.36, z = −5.2, p < .001). The model predicted that, on average, participants seated frontally across were looked at about 15% of the time, compared to 11% for laterally seated co-participants. Familiarity with the co-participant was significantly positively associated with face gaze proportion (β = .06, z = 2.1, p = .04), and the model predicted higher proportions towards familiar co-participants [familiarity rating 5 ≈ 16%; 95% CI (.14, .19)] than towards unfamiliar ones [rating 1 ≈ 13%; 95% CI (.11, .16)]. Participants' speaking-duration proportion was significantly positively associated with the amount of face gaze they received (β = 2.85, z = 5, p < .001): the more somebody spoke, the more they were looked at.

The model detected no statistical association between co-participant-specific face gaze proportions and the circumstance of previous dyadic conversation (p = .20). Gender configuration coefficients showed no significant change in co-participant gaze when both participants were male (p = .21) or female (p = .54) from when they were mixed (control). Dyadic movement synchrony with a specific co-participant was not significantly associated with face gaze proportions toward that co-participant (p = .39), providing no evidence for a relationship between gaze behavior and interpersonal movement synchrony.

Model fit was assessed using residual diagnostics. No evidence of over-dispersion (dispersion = .90, p = .68) or zero-inflation was found. To summarize, the proportion of time a participant spent looking at a co-participant's face was statistically associated with the co-participant's seating position, familiarity, and proportionate speaking-duration.

Exploratory analysis 3: mutual gaze

The third model considered as the dependent variable the proportions of mutual gaze between nested dyads (n = 72), i.e., the six dyadic axial configurations possible between the participants in each quartet. Recall that mutual gaze was detected whenever two participants' gaze cursors were simultaneously located within their respective facial AOIs. As such, mutual gaze proportion is a dyad-level measure and requires dyad-level versions of the predictors. Consequently, participants’ respective familiarity ratings and tracking ratios were averaged for each participant–co-participant pair. Speaking-duration proportion was not included because it does not have a meaningful dyad-level analogue that could be collapsed into a single quantitative measure. The predictors seating position, previous dyad status, gender configuration, and dyadic movement synchrony are dyad-level variables by design. Two random effect terms were included for each dyad and group. Results are shown in Table 3.

Table 3. Summary of parameter estimates and variance components from the GLM model of mutual gaze proportions.

Since the model used a logit link, coefficient estimates are expressed on the log-odds scale and should be interpreted in terms of direction and significance rather than magnitude.

Bold values indicate significance at α < .05.

Overall, the model-based estimated marginal mean of mutual gaze proportion was low at 3% [M^ = .03, 95% CI (.02, .04)], suggesting that dyads spent about 25 s of the average 14m07s quartet duration in mutual gaze. Mutual gaze proportions were associated with two covariates. Relative seating position significantly predicted mutual gaze: model-based predictions indicated that mutual gaze was observed at lower proportions (∼2%) for dyads seated laterally (β = −.55, z = −2.6, p < .01) than dyads facing each other frontally (∼4%). Average dyadic familiarity was positively associated with mutual gaze (β = .19, z = 2.02, p = .04), and model-predicted values indicated that familiar dyads [familiarity rating 5 ≈ 5%; 95% CI (.03, .09)] engaged in more mutual gaze than unfamiliar dyads [rating 1 ≈ 2%; 95% CI (.01, .05)]. In contrast to the previous models, average dyadic tracking ratio (p = .81) was not statistically associated with mutual gaze proportion. Neither gender configuration (female, p = .66; male, p = .19), dyadic movement synchrony (p = .72), nor previous dyadic coupling (p = .68) showed significant associations.

Random-effects variances indicated between-dyad and between-group differences in baseline mutual gaze proportions. Dispersion diagnostics (dispersion = 1.01, p = .86) confirmed robust model fit and support the reliability of the coefficient estimates. In summary, mutual gaze proportions were higher for frontally seated dyads than for laterally seated ones and increased with dyadic familiarity.

Discussion

This article developed a pipeline for automated mutual gaze detection that combines ArUco marker tracking, face detection, and a robust face identification script. The pipeline produces facial AOI data that is synchronized with eye-tracking data to compute participant- and dyad-specific social gaze events, which can be exported to ELAN for subsequent qualitative or for quantitative analysis. To evaluate the pipeline's validity and applicability in quantitative CA/IL research, we applied it to eye-tracking data recorded from groups of four people who were engaged spontaneous conversations and extracted co-participant-specific face gaze and mutual gaze data. The following sections therefore proceed in two parts. We first discuss the exploratory analysis as a case-in-point application and then return to the pipeline itself as the primary methodological contribution for future studies of mutual gaze in social interaction.

Discussion of the exploratory analysis

The aim of the exploratory analysis was to test the pipeline's usability for the exploration of gaze and mutual gaze duration between four participants in informal interaction. We derived three outcome measures from the pipeline results and, following previous work in Dirker et al. (in press), tested theoretically motivated predictors on proportions of individual face gazing, co-participant-specific face gaze, and mutual gaze in exploratory statistical analyses. The different outcomes can be related to the following research questions: (1) How does individuals' face gazing vary during spontaneous group conversation? (2) How do people distribute their gaze toward different co-participants in seated group interactions? (3) What factors may be associated with mutual gaze in multi-party interaction? The subsequent discussion develops hypotheses and interprets the results in the light of these questions.⁷

The first analysis modeled overall proportions of face gaze (vs. no-face gaze) as indicative of each participant's individual face gazing towards all three co-participants, irrespective of their identity, throughout the quartet interaction. The model fitted various predictors to explore how individual and situational characteristics may be related to these face gazing proportions. Results showed that participants did not exhibit different face gazing due to their age. Whether older or younger, individuals accumulated comparable face gazing proportions. While this is compatible with previous research (cf. Kleinke, 1986, p. 85), the distribution of age in our sample does not allow for generalization. Participants' average age was 27 ± 9 years and only 5 of the 48 participants were 36 or older. Further, these older individuals were found in only two groups, restricting inter-group variance and indicating that most co-participants were members of the same age group. Therefore, there may be too few observations to reliably estimate variation in individual face gazing along the age cline.

Notably, the analyses revealed two significant and complementary associations between participants' speaking duration and proportions of face gaze. First, for individual face gazing, participants' overall proportion of speaking duration was negatively correlated with their face gazing, suggesting that the more one spoke, the less they looked at others' faces. This was complemented by results from the second analysis (co-participant-specific face gaze), which showed that participants allocated significantly more gaze toward co-participants with higher proportions of speaking duration. In other words, the more somebody spoke, the more they were being looked at, and vice versa. These two results align with pioneering findings on the systematic differences between speaker and addressee gaze in managing participation (Goodwin, 1980a, 1984; Kendon, 1967, 1970; cf. Auer and Zima, 2021; Degutyte and Astell, 2021; Holler and Kendrick, 2015) and expand them to multi-party quartet interaction. Our analyses found stable speaking-duration effects besides other significant variables, suggesting that speaking is a reliable predictor of gaze allocation and aversion in our sample of spontaneous quartet conversations.

Participant familiarity showed a significant positive association with co-participant-specific face gaze and mutual gaze proportion. This suggests, first, that participants were more likely to direct face gaze at co-participants they knew better, and second, that these face gazes were reciprocated, thus establishing mutual gaze to larger degrees when the corresponding dyad partners’ average familiarity ratings were higher. This aligns with previous research on gaze addressee selection (cf. Kleinke, 1986, p. 88; Bennett et al., 2016 for conflicting results in a multi-party job interview context) and mutual gaze (Rubin, 1970, on romantic couples; Broz et al., 2012, for compatible results in dyadic conversations).

Further effects were detected for relative seating position. Participants in our sample consistently looked less at co-participants seated laterally than those seated frontally across from them, both in analyses of co-participant-specific face gaze and mutual gaze proportion. Previous research reported similar effects of seating position on mutual gaze (Argyle and Dean, 1965, comparing 90° adjacent seating with head-on dyads; Muirhead and Goldman, 1979, comparing side-by-side with frontal-facing dyads; Jokinen et al., 2013, analyzed side-by-side dyads conversing in a triad with a confidante; Blythe et al., 2018, for a variety of seating arrangements). Recall that participants were seated at a round table in a way that the default, comfortable seating orientation implied facing another, across-seated participant head-on. By contrast, gazing at co-participants seated laterally required participants to turn their head and/or trunk (cf. Schegloff, 1998, on body torque) to bring their interlocutors into the central visual field and to signal focused engagement [cf. Kendon, 1970, on establishing an “axis of interaction” (p. 113)]. Our empirical findings reflect the effort involved in such orientational adjustments and suggest that seating configuration is an important ecological factor in the emergence of unidirectional or mutual gaze. This underscores the relevance of different types of F-formations (Kendon, 1990) in seating arrangements (Kendon, 1970; Zima et al., 2025) and other embodied configurations (Kendon, 1990; Stukenbrock, 2020; Stukenbrock and Balantani, 2025) for the study of gaze in social interaction. Mutual gaze in side-by-side configurations is thus best understood as an interactional achievement—which is most evident in mobile side-by-side configurations (Auer and Laner, 2025; Stukenbrock, 2020, 2023)—, rather than a by-product of body or head orientation, and is therefore especially revealing of its interactional relevance.⁸ Notably, although the statistical analyses confirm a significant effect of seating position on gaze allocation in quartets, other covariates (e.g., speaking duration, familiarity) remain significant, suggesting that previously reported effects persist beyond seating arrangement effects.

Results do not support a relationship between gender configuration and gaze in our sample, neither for co-participant-specific face gaze nor mutual gaze. In comparison to mixed-gender dyads, neither female nor male pairs exhibited significantly different gaze proportions. This contrasts with gender differences found in previous studies (e.g., Argyle and Dean, 1965; Muirhead and Goldman, 1979; Lamb, 1981; Bennett et al., 2016; cf. the reviews by Kleinke, 1986, or Degutyte and Astell, 2021) but aligns with previous findings on dyadic gender configuration and movement synchrony of the same sample (Dirker et al., in press).

As far as we are aware, this study is the first to relate interpersonal movement synchrony to unilateral gaze behavior in spontaneous conversations between four participants. The present analyses provided no evidence for a statistical association between either of the two synchrony predictors, individual synchrony contribution and dyadic movement synchrony (collected in Dirker et al., in press), and the outcome variables individual face gazing, co-participant-specific face gaze and mutual gaze proportion. This suggests that, in our sample, movement synchrony did not correlate with gaze patterns once other predictors and grouping-level differences were accounted for. Pioneering empirical research (Kendon, 1970) made it probable that the degree to which an individual's and their co-participants' spontaneous body movements were temporally aligned would be related to the extent with which they look at them or they look at each other—especially considering recent empirical literature on links between interpersonal synchrony and social psychological concepts like affiliation (Sharon-David et al., 2019), shared attention (Wohltjen and Wheatley, 2021), and cooperation versus competition (Tschacher et al., 2014) (cf. Gordon et al., 2025, for review). It thus remains a question of future research whether, for instance, (a) individuals with higher synchrony contribution may use gaze to increase their conversational involvement (e.g., by attempting to claim and hold the floor) or (b) how movement synchrony of dyads nested in quartets may be related to mutual gaze versus unilateral gaze.

The circumstance of having engaged in a dyadic conversation shortly before the quartet interaction was examined as an additional dyad-level factor. The results did not indicate any differences in gaze proportions between dyads who had spoken alone (for about 14 min; see Dirker et al., in press) before rejoining the other group members, neither for co-participant-specific face gaze nor for mutual gaze proportion. In other words, no effect was found on how much participants looked at their previous dyad partner or on mutual gaze between them, even though a brief prior exchange might be expected to increase familiarity or momentary attunement in ways that promote mutual gaze, for instance, through increased personal common ground (Clark, 1996, 2021) resulting from a shared interactional history (Deppermann and Pekarek Doehler, 2021) at the micro-level. Evidence for a relationship between the duration of previous stranger dyads' spontaneous conversation and the proportion of mutual gaze has been reported by Cordell and McGahan (2004). From a statistical perspective, the absence of a prior-dyad effect might suggest that familiarity, as directly reported by the participants and a more fine-grained predictor (i.e., numeric rather than categorical), captured most of the variance that might otherwise have been attributable to the prior-dyad factor.⁹

Overall, the analysis of the gaze data of our quartet sample revealed that looking at others' faces, both unilaterally and mutually, was shaped primarily by speaking duration, seating configuration, and familiarity. Random-effects estimates indicated that participants differed markedly in their baseline tendency to attract gaze (ICC = .64; cf. Table 2), suggesting that some individuals are consistently more gaze-eliciting than others, independently of the measured predictors. This interpersonal variability aligns with long-standing observations in the literature. Kleinke (1986), for example, reviewed a wide range of quantitative and qualitative influences on mutual gaze, including cultural background, personality, and contextual or situational demands (e.g., Hoffmann et al., 2024). Although these factors were not considered in the present study, they may also have contributed to the unexplained variance captured by the models. Future research would benefit from systematically examining these additional determinants of gaze behavior, particularly in group settings where multiple social, spatial, and conversational dynamics intersect. Before moving on to the discussion of the pipeline, it is important to consider several caveats to the statistical analysis.

Limitations

Prior to the analyses, summary statistics and diagnostic plots indicated a positive association between measured face gazing proportions and the eye-tracking glasses' tracking ratio. The tracking ratio reflects the proportion of samples in which the device successfully recorded gaze data and thus serves as an immediate indicator of data availability and, indirectly, of recording quality. We accounted for this measurement dependency by including it as a covariate in all inferential models (cf. Tables 1–3). This adjustment reduced the risk that observed differences in gaze proportions reflected measurement artefacts rather than true behavioral variation. However, tracking ratio operates on a purely technological level and is therefore independent of participants' behavior or their subjective experience of the interaction. By contrast, all other predictors included in the models (e.g., speaking-duration proportions, synchrony contributions) index behavioral or interactional phenomena and thus operate on a different methodological level. Nonetheless, this does not eliminate the broader issue that quantitative measures of gaze in social interaction remain sensitive to variation in eye-tracking data quality. Studies relying on automated coding of gaze events should therefore aim to maintain tracking ratios that are both high and as consistent as possible across participants. Achieving this requires careful data-collection protocols that prioritize data quality over ecological validity concerns (cf. Holleman et al., 2020 for further discussion).

A further limitation of our study is the sample size and grouping structure of the data. We analyzed 12 of 20 quartets (48 participants), yielding 72 dyadic permutations nested within these quartets. Accordingly, statistical power is limited and estimates of between-group variability are less precise. Small effects may therefore remain undetected and variance components should be interpreted with caution. Replication in larger samples is desirable.

Despite this limitation, predictors identified by theory and prior research as most relevant—spatial co-orientation, speaking duration, and familiarity—were robustly significant in our models. This supports the substantive validity of the analyses and suggests that the observed patterns reflect meaningful social processes rather than sampling variability.

Discussion of the pipeline

Temporal synchronization

Synchronizing eye-tracking data with scene video recordings presents several challenges. The two data streams operate at different sampling rates and increment at different intervals, and they are controlled by separate hardware systems, each of which may introduce timing irregularities (e.g., dropped samples or frames). In addition, the recordings run on different internal clocks, which was also noted by Niehorster et al. (2025: 4), and we found video recording to be slightly delayed to pupil data recording in the Glasses API. This limits the precision with which the two streams can be synchronized, and converting eye tracking timestamps into scene video frame counts therefore involves some degree of uncertainty. These timing discrepancies may accumulate over the course of a recording, increasing the likelihood that frame and gaze samples drift slightly out of alignment toward the end.¹⁰

As described in Step 2 of the methodology chapter, we performed temporal synchronization of gaze and scene video data by manual identification of start and end synchronization points in video frame timestamps and matching them to eye-tracking timestamps. More advanced synchronization point retrieval options exist, including sending repeated simultaneous voltage pulses to all recording units (TobiiPro, 2021, p. 59), triggering synchronized starts of multiple recording units via a shared network connection (Matar et al., 2025), or revealing a unique ArUco marker to all participants simultaneously at on- and offset of the critical interaction and using the frames in which it was first and last detected as anchors (Niehorster et al., 2025). These methods vary in technological complexity, precision, and dependence on specific software systems.

Spatial synchronization

Spatial synchronization, the mapping of positional gaze data to facial AOI data, is affected by gaze data quality. Even small shifts of the eye-tracking head unit, caused by natural behaviors such as head-turning, speaking, laughing, or facial expressions, reduce tracking accuracy or cause data loss. Prior work has documented how small amounts of head-unit slippage can substantially degrade gaze estimation accuracy in several commercially available mobile eye trackers (Niehorster et al., 2020). As a result, gaze shifts to another person's face may be missed entirely (false negatives) or registered inaccurately (false positives). In line with this, statistical analyses confirmed that proportion of detected gaze events was positively correlated with tracking ratio. These issues may accumulate over time, making longer recordings particularly vulnerable to spatial desynchronization and consequently event detection errors.

Scene video quality also contributes to spatial synchronization problems, particularly when ArUco markers or faces cannot be reliably detected. Incomplete tracking often resulted from suboptimal lighting conditions, motion blur, or from gaze cursor overlay obscuring parts of the image. Since the RetinaFace model demonstrated validity on benchmark datasets such as WIDER FACE (Deng et al., 2019), we assumed that it would perform reliably, but face detection performance may degrade under real-world recording conditions. Other researchers employed alternative computer-vision tools (OpenFace by Hessels et al., 2018; OpenPose by Jongerius et al., 2021; and Callemein et al., 2019; YOLOv2 by Callemein et al., 2019). Future work may compare these approaches to determine which algorithms remain most accurate when scene video image quality is less than ideal.

The face identification procedure described in Step 1.3, which links face coordinates to ArUco marker positions, still requires manual verification to ensure accuracy. This need for human intervention decreases under more controlled conditions. For example, maintaining constant luminance improves the detection of faces and markers; printing markers in larger formats and in their default white-on-black design enhances detection robustness across angles and distances. Fixing participant chairs in identical positions stabilizes participants’ FOVs across recordings, reducing fluctuation in marker-relative face locations, improving automated identifiability and minimizing manual fine-tuning. Recalibrating the eye-tracking glasses during data collection further counteracts slippage-induced gaze estimation errors and shifts in the field of view. Striking a balance between ecological validity and condition control is therefore crucial to maximize both data quality and naturalistic behavior.

Adaptive error margin

A common strategy to remedy gaze estimation inaccuracy in mobile eye-tracking studies of mutual gaze has been to “not distinguish between facial features and instead […] make cruder distinctions, such as whether gaze was directed on the face […] or somewhere else” (Valtakari et al., 2021, p. 1,599). Accordingly, this pipeline added an error margin (cf. Step 2.1) around the circular facial AOI to classify mutual gaze. The size of this added error margin, here set to half of the facial bounding box' circumradius, demands further discussion.

Lappi (2015, p. 15) argues that the “size of a useful AOI depends on the positional accuracy” of the eye-tracking device and should not be an arbitrary heuristic. Jongerius et al. (2021) included an adjustable error margin. Matar et al. (2025) chose a triple of the average gaze estimation error of their eye-tracking devices. Gaze estimation error is a technical index of the recording situation and varies over time (Niehorster et al., 2020). Such indices are often not registered by mobile eye-tracking systems since neither calibration validation nor sporadic drift checks are implemented. Hence, several research groups (MacInnes et al., 2018; Mayrand et al., 2023; Niehorster et al., 2024) designed their own validation procedures and measured spatial accuracy by having participants fixate predefined targets on a standardized display and computing the deviation between the subjectively reported and estimated gaze positions. Such deviation measures could be used to compute subject-specific error margins.

Beyond the hardware constraints, the area to be considered for face gaze depends on the physical distance between participant and co-participant, especially when their mutual perception of eye contact is the object of investigation. Research in psychology demonstrated that the recognition of mutual gaze depends on viewing distance (Gibson and Pick, 1963). It was found that gaze toward the facial periphery was more frequently judged to constitute eye contact with increasing distance (Lord and Haith, 1974). Consequently, the larger the distance between participants, the larger the area where their gaze is perceived by others as directed at them. Co-participants may judge gaze toward their eyes or mouth, their face, or their entire body as eye contact. Therefore, the error margin should be larger as a factor of viewing distance. Given that participants experience affective reactions to eye contact only when they perceive it as such (Hietanen, 2018; Honma, 2013, for review), the most accurate computation of a distance factor involves individual testing of participants.

The error margin applied in the present analyses was distance-insensitive and, although it assigned the frontally seated co-participant the smallest facial AOI, it did not appear to impede gaze event detection. Most unilateral face gaze and mutual gaze events involved the person seated directly across, suggesting that the smaller AOI was not inherently detrimental. Gaze at the participant seated across involves a fixation with minimal head and eyeball rotation, facilitating pupil detectability and, therefore, more consistent gaze estimation compared to extreme lateral gaze angles. Nonetheless, the heuristic remains problematic in situations where participants direct their gaze toward the frontal partner while their heads are turned, as the combination of extreme gaze angles and a smaller AOI increases the risk of false negatives. These considerations reinforce the need for recalibration before critical interactions and for a technically and phenomenologically adaptive computation of facial AOI error margins, as outlined above.

Flexibility and adaptability

In contrast to the caveats, the following subsection highlights the strengths of the pipeline.

The present data was collected using TobiiPro hardware, and raw gaze and video data formats may differ across manufacturers. Because the pipeline operates entirely on open-source applications and accepts any mobile eye-tracking output in scene video coordinates, it is applicable beyond TobiiPro systems.¹¹ By relying on computer-vision applications, the pipeline eliminates the need for labor-intensive manual coding, substantially reducing the time, cost, and specialized expertise typically required for frame-by-frame annotation. Automated processing also ensures consistent application of detection criteria across thousands of frames, avoiding the variability inherent in human annotators. Integrated development environments for both R and Python, along with required packages and libraries, are freely available, making the pipeline a cost-efficient alternative to commercial software. Its local execution avoids cloud-based inference or other mechanisms that may raise concerns under data-protection regulations, while still allowing researchers to modify and extend the code as needed.

Like other script- or GUI-based solutions, the pipeline requires a degree of technical expertise. Users need basic programming skills (e.g., setting up a Python or R environment, installing and troubleshooting package dependencies, loading and processing data tables, and adjusting script parameters). A working grasp of the underlying geometry is also necessary to understand the individual processing steps, which may lack in research teams. Nevertheless, the annotated code, available documentation, and transparent processing steps help flatten the learning curve, and many of the required skills can be acquired quickly, enabling researchers to apply the pipeline to their own data.

Reliable performance of the pipeline hinges on the quality of both eye and scene video data, and data quality issues are to be expected in in-the-wild multi-party interaction. Therefore, as a sanity check before gaze data synchronization, the pipeline compares how many of the current participant's samples with detected faces remain unlabeled. In this study, we proceeded only once all detected faces had been labeled. This maximalist approach ensured that all co-participants' faces in our seated-quartet configuration were identified and served as input to social gaze event quantification. Additional post-processing may be applied, such as imposing minimum thresholds for mutual gaze durations (cf. Rühlemann and Ptak, 2023, p. 33). For qualitative analyses of gaze in social interaction, most CA/IL researchers rely on ELAN transcriptions, and the pipeline provides ELAN-compatible output aligned with source video time, allowing analysis to begin immediately after import.

The ArUco-marker-based identification method further enables adaptation of the pipeline to larger multi-party configurations or to other modalities (e.g., computer-vision detected participant-specific gestures, Ripperda et al., 2020) without adding substantial complexity. The main consideration is thorough marker placement: in seated configurations, markers can be positioned between participants, whereas in mobile configurations wearable ArUco lanyards or clothing patches could be used.

In summary, the pipeline operates with open-source tools, accepts any mobile eye-tracking output in scene video coordinates, and implements sanity checks and validity controls to ensure reliable face detection and social gaze event extraction. It produces ELAN-compatible output for immediate analysis. The pipeline is adaptable to different settings and larger multi-party configurations and markedly more time-efficient than human coding, as it processes the same raw gaze data in a fully automated manner that would require hours of frame-by-frame inspection by human coders.

The previous discussion highlighted the advantages of this pipeline for studies of gaze in social interaction. However, the present implementation would benefit from formal validation of the reliability of social gaze event extraction in the future. At this stage, additional human validation of the final ELAN output was not included because TobiiProLab's scene video gaze overlays are, by default, based on filtered, aggregated gaze events (i.e., fixations) rather than the raw 50 Hz data used for the pipeline. This filtering process turns the raw gaze data into aggregated fixation events while suppressing small fixational and non-fixational eye movements. Therefore, the reliability of the gaze overlay cannot be established, and human coders of the scene videos would be assessing pre-processed gaze data that differs from the raw data (i.e., ground truth) underlying the pipeline.

Nonetheless, several aspects indicate that the pipeline produces robust results. First, human validity checks are included at multiple processing steps: conditional statements for co-participant identification are adjusted and verified by the script runner using schematic scene video screenshots. Second, cumulative samples of detected faces are compared with ultimately identified face samples before and after processing to ensure that all samples with facial coordinates are labeled correctly. While this approach renders the pipeline more labor-intensive than fully automated alternatives, the continuous human validation minimizes misclassification of co-participants and ensures that only verified face labels are used for subsequent gaze event extraction. Third, statistical analyses of the pipeline output align with and confirm previous findings. The gaze data were tested for correlation with independently human-coded speech duration proportions derived from ELAN transcripts, which were produced by trained research assistants under supervision of experienced CA researchers. Prior work has shown systematic relationships between speaking and gaze behavior in conversation, and the pipeline's extracted gaze measures reproduce these well-established patterns with statistically significant effect sizes. In other words, two independent measurement streams, the detected gaze events and human-coded speech, converged in theoretically predicted ways, providing evidence that the pipeline captures meaningful gaze behavior, while ongoing human validation further strengthens its reliability in “in-the-wild” eye-tracking data.

Conclusion

This article introduced a novel, mobile eye-tracking data-processing pipeline that automates the detection of unilateral and mutual gaze in complex multi-party interactions by combining ArUco marker tracking with face detection. The pipeline was applied to informal quartet interactions to demonstrate its practical usability. An exploratory analysis showed that speaking proportion, seating position, and participant familiarity were the main factors associated with gaze allocation to co-participants in this sample. No evidence was found that age, gender configuration, or interpersonal movement synchrony were statistically related to these gaze patterns. Although challenges related to temporal and spatial synchronization persist in naturalistic settings, the proposed methodology offers a robust, high-fidelity alternative to traditional manual coding. By providing transparent processing steps and ELAN-compatible outputs, the pipeline creates a bridge between quantitative and qualitative analyses of social gaze, thereby supporting mixed-methods approaches in multimodal social interaction research.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomm.2026.1857066/full#supplementary-material

Statistical analysis Supplementary material at the source

Pipeline scripts and sample data Supplementary material at the source

Footnotes

Edited by: Bert Oben, KU Leuven, Belgium

Reviewed by: Anastasia Bauer, University of Cologne, Germany

Niilo Valtakari, Utrecht University, Netherlands

Footnotes

¹In a validation study, Jongerius et al. (2021) compared software-annotated face-gaze data from the first sixty seconds of seven eye-tracking videos to annotations generated from AOIs drawn by human coders in TobiiProLab. The gaze data relative to these human-drawn AOIs was then compared to the automatically extracted face-gaze data. Agreement between human coders and the algorithm was very high (Cohen's kappa ≥ 88%), indicating that the software detects face-gaze at a level comparable to human annotators.

²The pipeline can in principle be applied to dyadic interaction, where only one co-participant's face is detectable in each scene video. The visible partner need not be distinguished from additional faces, which substantially simplifies processing, as the face identification procedure (see Step 1.3) becomes unnecessary.

³Image-processing algorithms identify ArUco markers by detecting the grid patterns and matching the extracted bit configurations against predefined dictionaries. Each unique arrangement corresponds to a specific ID, enabling unambiguous marker identification. ArUco markers are widely used in robotics and computer vision applications, e.g., to facilitate camera pose estimation, object localization, and distance measurement in navigation tasks. The ArUco markers for this study were generated from a website, see https://chev.me/arucogen/, last on Feb. 28th, 2026.

⁴Referring to Tschacher et al.'s (2021) gaze-cursor tracking, we chose to leave open the possibility to detect ArUco markers by color.

⁵Up to date performance measures are available for the WIDER FACE validation dataset (Yang et al., 2016) at http://shuoyang1213.me/WIDERFACE/WiderFace_Results.html (last on Feb. 28th, 2026).

⁶Each participant can be paired with any of the other three, yielding six unique dyads nested within a quartet.

⁷Linguistic questions are not addressed here and will be examined in subsequent investigations.

⁸Considering the strong correlation between gaze and seating position, we deem it an interesting endeavor of future qualitative research to examine potential effects of seating position on floor management.

⁹In the present models, familiarity did not correlate with previous dyad status, and no multicollinearity was detected. Relatedly, in a previous study on movement synchrony, we also found no evidence for effects of previous dyad status on movement synchrony of dyads nested in quartets (Dirker et al., in press).

¹⁰However, see the TobiiPro SDK website on "drift compensation" (https://developer.tobiipro.com/commonconcepts/timestamp-and-timing.html, last on 15^th April, 2026).

¹¹Note that the export of raw data was handled with TobiiPro Lab out of convenience. An open-access software development kit available for TobiiPro Glasses 3 (see https://github.com/tobii/tobii-glasses3-sdk since Feb. 2024, last on Feb. 28th, 2026) allows researchers to extract raw gaze data from .json-files accessible on the SD card, avoiding the cost for the manufacturer's software.