Predicting female and male participants' contact-exchange decisions from time-segmented multimodal behavioral features in speed dating

1 Introduction

The speed dating paradigm is widely used to study romantic interaction (Finkel et al., 2007; Finkel and Eastwick, 2008). It offers a relatively natural yet structured setting. Unfamiliar partners engage in short conversations and proceed to contact exchange only when both parties express interest. This framework is useful because it lets researchers examine in-conversation decision processes that are difficult to recover from static attributes alone, such as profile information or post hoc self-reports (Finkel et al., 2007). However, many existing studies of contact-exchange and attraction outcomes still rely mainly on participant attributes, pre-existing preferences, post-interaction questionnaires, or only a limited subset of behavioral cues. As a result, it remains insufficiently understood how multimodal behaviors during the conversation itself, and their temporal changes, relate to those outcomes (Fisman et al., 2006; Houser et al., 2008; Joel et al., 2017; Ali et al., 2018). This makes it difficult to explain, on the basis of observable evidence, which behaviors are associated with favorable outcomes. This issue is also practically relevant. A recent Japanese governmental survey reported demand for support concerning communication methods, relationship development, and counseling. This demand underscores the practical value of clarifying observable behavioral patterns in face-to-face interaction (Children and Families Agency, Japan, 2024).

Our goal is to clarify, in interpretable terms, the behavioral patterns associated with contact-exchange outcomes. To this end, we test whether interpretable multimodal features extracted from 1-min segments of a 10-min interaction can predict contact-exchange outcomes. We then analyze model-derived feature importance by behavioral feature, modality, and time interval. The dataset separately records contact-exchange decisions made by female and male participants. We therefore modeled female and male participants' contact-exchange decisions separately.

This study is organized around three contributions.

• We organized visual, audio, linguistic, and interaction features by 1-min interval.
• We evaluated separate Random Forest models for contact-exchange decisions made by female and male participants using participant-level cross-validation.
• We compared feature-set performance and analyzed feature importance by behavioral feature and time interval for contact-exchange decisions made by female and male participants.

2 Related work

2.1 Analysis of speed dating

Many speed dating studies analyze attraction and partner choice through self-reported preferences, personality traits, and partner evaluations. These studies help explain what kinds of partners people prefer and how such preferences differ across individuals and situations (Finkel et al., 2007; Fisman et al., 2006; Valentine et al., 2020; Eastwick and Finkel, 2008). Related studies have further examined perceived similarity, attachment-related tendencies, longer-term relational outcomes, and personality-linked attractiveness. Together, this work indicates that early mate selection is shaped by multiple psychological factors rather than by a single preference dimension (Tidwell et al., 2013; McClure et al., 2010; Asendorpf et al., 2011; Jauk et al., 2016). These studies are important for theory building. However, they are less directly suited to identifying interpretable behavioral patterns in observed interaction, because they do not fully describe how behavior changes during an actual conversation. Prior studies based on self-reported indicators can also be divided by measurement timing. Some use measures collected before the event, such as preferences and personality traits. Others use measures collected after the interaction, such as impressions and partner evaluations. While both approaches have been informative, they are also sensitive to contextual and situational factors, which makes it difficult to treat them as direct descriptions of behavior unfolding in real time. (Joel et al., 2017) showed that initial romantic desire is hard to predict from static descriptors alone, even when a wide range of personal characteristics is available. Their result reinforces the need to study what actually happens during the interaction rather than relying only on what participants say they want or report afterward. Other studies have shown that speed dating outcomes are shaped by situational and interpersonal factors. These include sequential contrast effects and the affective presence of the interaction partner, again suggesting that favorable outcomes are not explained by static traits alone (Bhargava and Fisman, 2014; Berrios et al., 2015).

At the same time, some studies focus more directly on conversational behavior itself. This line of work includes prediction from brief-observation judgments (Houser et al., 2007), communication patterns associated with speed dating success (Houser et al., 2008), spoken interaction style and flirting-related cues (Jurafsky et al., 2009; Ranganath et al., 2009, 2013), and video-based or other non-verbal behavior in recorded interactions (Veenstra and Hung, 2011; Ali et al., 2018). Even in this line of research, however, many studies either treat the entire conversation as a single unit or focus on one or only a few modalities. As a result, they provide limited information about when relevant behaviors appear and how their contributions differ across modalities. This limits how clearly outcome-related behaviors can be described in a readily understandable form.

2.2 Multimodal behavior analysis

Multimodal behavior analysis combines information from multiple channels. Facial behavior analysis, acoustic feature extraction, morphological analysis, and sentence-level semantic representations can describe human interaction from complementary perspectives (Beyan et al., 2023; Baltrušaitis et al., 2016; Eyben et al., 2010; Kudo et al., 2004; Reimers and Gurevych, 2019). This is particularly relevant to romantic interaction, where favorable or unfavorable impressions may depend on combinations of behaviors across modalities rather than on any single cue alone (Roth et al., 2021). Beyond romantic settings, related work in dyadic negotiation and interview scenarios has shown that mutual non-verbal behavior can predict immediate reactions and social evaluations such as hirability. This supports the value of modeling interaction patterns rather than analyzing behaviors one by one (Park et al., 2012, 2013; Nguyen et al., 2014).

At the same time, heavy reliance on complex latent representations can make it harder to connect the analysis to behavioral features that remain readily interpretable to humans. For this reason, we use embeddings only in a limited supporting role and rely primarily on interpretable behavioral features as the main unit of analysis. This allows facial actions, audio dynamics, lexical content, and interactional coordination to be discussed in human-readable terms and compared across time windows.

2.3 Motivation for separate analyses of female and male participants' decisions

Previous work on speed dating has repeatedly suggested that preferences and decision processes may differ between women and men, implying that the same behavioral signal may not play the same role for both groups (Fisman et al., 2006; Eastwick and Finkel, 2008). Gender differences have also been reported in the ability to read emotion from subtle facial expressions, although the pattern can vary with expression intensity and task demands (Hoffmann et al., 2010; Hong et al., 2015). Taken together, these findings motivate separate analyses of female and male participants' decisions rather than treating participant category only as a control variable. Accordingly, in the present study we model female and male participants' contact-exchange decisions separately rather than averaging them in a pooled analysis. This design supports descriptive examination of the model-derived behavioral patterns within each decision group while preserving explicit time segmentation and interpretable multimodal features.

3 Methodology

3.1 Dataset

We used the Multi-Modal Speed Dating (MMSD) corpus constructed by (Ishii et al., 2023). The corpus consists of 625 speed dating dialogue instances organized into 25 session units and involving 147 Japanese participants (75 female and 72 male participants). Participants self-reported their sex as female or male, consistent with their biological sex. Participants ranged in age from their teens to their sixties, with a mean age of 31.9 years (SD = 8.6). The female participants had a mean age of 31.35 years, a median of 29 years, and a range of 19–60 years. The male participants had a mean age of 32.39 years, a median of 30 years, and a range of 19–58 years. Numerical counts by age range are reported in Table 1.

Table 1. Participant counts by age range for female and male participants.

Each session unit includes a 5-min self-introduction segment followed by multiple 10-min dialogue instances. From these 10-min dialogue instances, we used the video recordings, audio recordings, transcripts, and labels indicating whether each participant wished to exchange contact information. Video was recorded from a slightly elevated frontal viewpoint capturing the upper body, and audio was recorded using microphones worn by the participants. The recording environment is illustrated in Figure 1.

The dialogues in this corpus were collected during actual speed dating events intended for matching, and the experimenters did not control the flow of conversation or the roles of the participants. Participants were aware that the event was a real speed dating event and interacted freely with one another. Of the 625 source dialogue instances, 619 had observed decisions from both participants; in the remaining six, both decisions were missing. After excluding these six instances, Table 2 shows the distribution of contact-exchange decisions, with the 619 complete decision pairs used as the denominator for the percentages.

Table 2. Cross-tabulation of contact-exchange decisions by male and female participants.

3.2 Feature extraction

This subsection describes how we extracted multimodal features from participant behavior in each temporal segment of the speed dating interaction. Drawing on prior work that used minute-scale temporal segments and 1-min behavioral slices (Ali et al., 2018; Murphy, 2005), we divided each 10-min speed dating dialogue into 10 consecutive 1-min segments. Because interpretability is central to this study, we limited feature extraction to relatively transparent features. Using this temporal segmentation and the interpretable features, we examined whether observable behavioral patterns were informative for predicting contact-exchange decisions and which behaviors were associated with those decisions. The features extracted in this study are summarized in Table 3.

Table 3. Feature sets extracted from each 1-min segment.

3.2.1 Visual features

This subsection describes how visual features were extracted from participant behavior. For the video data in the MMSD corpus, we used the facial-image processing toolkit OpenFace (Baltrušaitis et al., 2016) to extract features related to head and facial behavior. Feature extraction was carried out at 60 frames per second, matching the video frame rate. As visual features, we extracted measures related to head orientation, gaze direction, and facial Action Units (AUs), which represent basic units of facial muscle activity (Ekman and Friesen, 1978). For head orientation, we used the rotation angles around the x-, y-, and z-axes (pose_Rx, pose_Ry, and pose_Rz), and for gaze direction we used the horizontal and vertical rotation angles (gaze_angle_x and gaze_angle_y). For head, gaze, and the AU intensity signals listed in Table 4, we computed the variance, median, 10th percentile, and 90th percentile within each segment. Participant-side statistics were calculated from frames with success == 1. As shown in Figure 2, the coordinate system is defined from the camera viewpoint: x from left to right, y from bottom to top, and z from front to back.

Table 4. Facial action units used as visual features.

To capture interaction between the participants, we additionally introduced features related to synchrony in smiling behavior and head orientation. This choice is consistent with prior work showing that coordinated non-verbal behavior can be informative for interpersonal reactions and social outcomes in dyadic settings (Park et al., 2012, 2013; Nguyen et al., 2014). This choice also reflects the view that interaction dynamics are an important part of human behavior analysis (Beyan et al., 2023). For smiling behavior, we computed correlation coefficients between the two participants for the intensities of AU06 and AU12, which are core components of smiling. In addition, for head movement we computed between-participant correlation coefficients for the head-rotation angles around the x-, y-, and z-axes. The dyadic OpenFace correlations used the available frames without applying thresholds based on the success or confidence flags.

3.2.2 Audio features

This subsection describes how audio features were extracted. From the audio data of each participant, we extracted representative audio and speech-related features using the audio-processing toolkit openSMILE (Eyben et al., 2010). For each target segment, we first extracted the speech intervals of each speaker. We then computed the statistics listed in Table 5 for the audio features shown in Table 6.

Table 5. Summary statistics computed for each audio feature.

Table 6. Audio feature types extracted with openSMILE.

To describe within-interval co-variation between participants in the audio channel, we calculated a windowed acoustic correlation for each feature listed in Table 6, drawing on the windowing and correlation procedures used in the Time-Aligned Moving Average (TAMA) approach (Kousidis et al., 2009; Vaughan, 2011). Each 1-min interval was divided into five overlapping 20-s windows with a 10-s step. For each participant and window, openSMILE acoustic-feature values corresponding to that participant's speech were selected and averaged separately for each feature. We then calculated the Pearson correlation between the two participants' five window-level values. When too few valid paired window-level values were available or either series was constant, the Pearson correlation could not be calculated; these cases were treated as missing. This calculation did not use speaker-wise full-conversation normalization: only observations within the relevant 1-min interval were used, so observations from preceding or subsequent intervals did not contribute to that interval's correlation feature. Non-finite acoustic frames were excluded. A segment was treated as missing when it contained no valid speech or acoustic frame. Invalid fundamental-frequency frames were excluded, and the corresponding feature was treated as missing when no valid value remained. We also included the absolute difference in speech rate between participants as an interaction feature. Speech rate was calculated by dividing each utterance duration by the number of morae computed with UniDic and averaging the values with utterance-duration weights within each segment.

3.2.3 Linguistic features

This subsection describes how linguistic features were extracted. We performed morphological analysis on participant utterances and extracted linguistic features based on part-of-speech frequencies. For morphological analysis, we used the Japanese morphological analyzer MeCab (Kudo et al., 2004). Each word extracted from the utterances was classified into 13 part-of-speech categories, and the occurrence count of each category was divided by the total number of words to obtain its relative frequency. In addition, we introduced the absolute differences in these part-of-speech frequencies between the two participants as interaction features. This part-of-speech grouping is based on the major categories in the UniDic tag system, excluding symbols, auxiliary symbols, and blanks (Den et al., 2008).

To capture similarity in utterance content between participants, we additionally introduced linguistic features based on semantic similarity. For this purpose, we used a pre-trained Sentence-BERT model (Reimers and Gurevych, 2019, 2020) and transformed each utterance into a 384-dimensional vector. Similarity between utterances was computed as cosine similarity between these vectors. To compute similarity to the partner within a time segment, we compared each utterance with the partner's utterances that had started within the previous 30 s and used the maximum similarity as the value for that utterance. When no counterpart utterance satisfied the 30-s condition, that utterance was excluded from the interval-level statistics. The same procedure was applied to the other speaker, and quartiles of the resulting similarity distribution were used as speaker-level features. These features were computed as follows.

• Transform each utterance into a 384-dimensional vector using Sentence-BERT.
• For each utterance of one speaker, compute cosine similarity to the partner's utterances that started within the previous 30 s.
• Define the maximum of those similarities as the similarity value for that utterance.
• Repeat the same procedure for the other speaker.
• Compute quartiles of the similarity distribution for each speaker and use them as features.

3.3 Model construction

This subsection describes how machine-learning models were constructed to predict whether participants would exchange contact information.

3.3.1 Model overview

This subsection outlines the prediction models. The overall modeling pipeline is illustrated in Figure 3. We constructed separate models for contact-exchange decisions made by female and male participants. Hereafter, these are referred to as the female and male models, respectively. We used interpretable features to examine behavioral features related to prediction. We used SHAP values for feature selection and feature-importance analysis.

Prediction target

The prediction target was each participant's decision about whether to exchange contact information, as described in the dataset subsection. The task was therefore formulated as a binary classification problem, where class 1 denotes contact exchange and class 0 denotes no contact exchange.

Predictors

The predictors were the visual, audio, linguistic, and interaction features extracted in the feature-extraction subsection from each of the ten 1-min segments of a 10-min speed dating dialogue.

Model

We used Random Forest as the learning algorithm. For each outer fold, we used only the outer-training data to select Random Forest hyperparameters from 1,920 configurations based on inner-cross-validation weighted F1 and to perform SHAP-based feature selection. We fitted the final model with the selected hyperparameters and feature subset and evaluated it on the outer-test data. Further procedural details are given in Section 3.3.2, and the complete search grid and fixed settings are listed in Supplementary Section S3 (Supplementary Table S3).

3.3.2 Evaluation protocol

The female and male models were evaluated separately using a fully nested participant-level leave-one-participant-out (LOPO) procedure. The 625 dialogues involved 147 participants (75 female and 72 male participants); outer evaluation used 75 folds for the female model and 72 for the male model. In each outer fold, all dialogues involving one participant were assigned to the final test set, and dialogues involving any of that participant's counterparts were excluded from the outer-training set.

Within each outer-training set, we applied inner participant-level LOPO using the same counterpart-exclusion rule. The number of inner folds ranged from 68 to 74 for the female model and from 65 to 71 for the male model. We evaluated all 1,920 hyperparameter configurations using weighted F1 in each inner fold and selected the configuration with the highest equally weighted mean across folds.

We then iterated SHAP-based feature removal within the same inner LOPO. At each iteration, models were fitted to the inner-training data, up to 100 training dialogues were used as reference data for the SHAP calculation, and mean absolute SHAP values were calculated on the inner-validation data. Features at or below the fifth percentile of importance were removed while retaining at least 10 features, and the subset with the highest inner mean weighted F1 was selected. The search stopped after three consecutive iterations more than 0.01 below the best score.

The selected hyperparameters and feature subset were used to fit the final model on the complete outer-training data, after which the outer-test data (kept separate from tuning, feature selection, and stopping decisions) were evaluated once.

Preprocessing used complete-case analysis for each feature configuration. Dialogues missing any required feature were excluded from the corresponding configuration; no imputation, scale transformation, or outlier processing was applied. After complete-case filtering, 613 dialogues remained for the all-feature configuration, 617 for visual features, 615 for audio features, and 616 for linguistic features. Final performance was calculated by pooling the LOPO out-of-fold predictions. To quantify uncertainty in the performance metrics, we calculated 95% intervals from 10,000 bootstrap replicates with seed 0, resampling the test participants as clusters.

As a baseline, we randomly predicted Exchange or Not Exchange according to the class proportions in each outer-training fold. To quantify uncertainty in the weighted-F1 differences between Random Forest and the baseline, we performed 10,000 bootstrap resamples of test participants, keeping each participant's dialogues together. We calculated 95% intervals and two-sided bootstrap p-values for the differences and applied Holm adjustment to the p-values for the eight comparisons. Supplementary Section S3 provides the settings for the additional baseline comparisons and model analyses, together with the software specifications.

3.3.3 SHAP feature-importance and time-interval analysis

For each outer fold, we calculated SHAP values for the held-out participant's test dialogues using only the features selected in that fold. One hundred dialogues sampled with seed 0 from the corresponding outer-training data were used as reference data. Within each fold, feature importance was defined as the mean absolute SHAP value across both prediction classes and all test dialogues. To combine results across folds, a feature not selected in a fold was assigned zero importance, and the fold-level values were averaged with equal weights. To summarize variation across folds, 95% bootstrap intervals were calculated from 10,000 resamples of the outer folds.

To examine how importance was distributed over the conversation, we summed absolute SHAP values for features belonging to each 1-min interval and averaged the interval totals within each fold. The 10 interval totals were converted to percentages summing to 100% within each fold and then averaged with equal weights across folds. To assess similarity between adjacent 1-min intervals, Pearson correlations were calculated between corresponding features using dialogues with available values in both intervals.

4 Experiments and results

4.1 Prediction results

This subsection presents prediction performance for the four feature configurations of the female and male models. Table 7 reports weighted F1, macro F1, balanced accuracy, and Matthews correlation coefficient (MCC). Class-specific precision, recall, F1, and confusion matrices for all eight configurations are provided in Supplementary Section S4 (Supplementary Table S9). For the all-feature female model and visual-feature male model, Supplementary Table S10 reports metric estimates and participant-cluster bootstrap 95% intervals, including ROC-AUC and Exchange average precision.

Table 7. Test performance under nested participant-level LOPO.

Bold values indicate the performance values of the best-performing model for the male and female models, respectively.

For the female model, the all-feature configuration had the highest weighted F1 (0.6248). Its F1 was 0.7647 for Not Exchange and 0.2987 for Exchange, and Exchange recall was 0.2500. ROC-AUC was 0.5818 and Exchange average precision was 0.3534. For the male model, the visual-feature configuration had the highest weighted F1 (0.5379); its class-specific F1 values were 0.5336 and 0.5425, ROC-AUC was 0.5560, and Exchange average precision was 0.5310. The 95% bootstrap intervals for weighted F1, obtained by resampling test participants, were 0.5589–0.6919 for the all-feature female model and 0.494–0.579 for the visual-feature male model. Participant-cluster bootstrap 95% intervals for balanced accuracy, MCC, ROC-AUC, and Exchange average precision are reported in Supplementary Table S10. For both the all-feature female model and the visual-feature male model, the balanced-accuracy intervals included 0.5 and the MCC intervals included zero.

Table 8 compares each Random Forest configuration with random predictions based on the class proportions in the corresponding outer-training folds.

Table 8. Weighted F1 comparison between random forest and class-proportion random prediction.

Random Forest weighted F1 was numerically higher than the mean random baseline in all eight configurations. However, every 95% interval for the difference included zero, so no configuration showed a clear improvement over random prediction after accounting for variation among participants. Because the all-feature configuration had the highest weighted F1 for the female model, we performed an additional comparison using the 613 dialogues shared by all four configurations. Weighted F1 was 0.6248 for all features, 0.6156 for linguistic features, 0.6137 for visual features, and 0.5919 for audio features. The differences between the all-feature and visual, audio, and linguistic configurations were 0.0111, 0.0329, and 0.0093. The corresponding participant-level bootstrap 95% intervals were –0.0303 to 0.0496, –0.0111 to 0.0736, and –0.0372 to 0.0534; all included zero, so the differences among configurations were unclear.

Feature-retention results are reported in Supplementary Table S6, and the conditional feature-removal ablation and comparison with L2 logistic regression are reported in Supplementary Table S7. The diagnostic training-set-size analysis conducted with fixed fold-specific settings is reported in Supplementary Table S8, Supplementary Figure S1, and alternative baselines for the all-feature female model and visual-feature male model are reported in Supplementary Table S11. The 100% condition in this diagnostic analysis is not a reproduction of the original fully nested SHAP feature-selection pipeline.

4.2 Feature importance analysis

This subsection analyzes SHAP-based feature importance. Of the four evaluated feature sets, the all-feature model yielded the highest observed weighted F1 for female participants' decisions, whereas the visual-feature model yielded the highest observed weighted F1 for male participants' decisions. We therefore focused the SHAP-based feature-importance analysis on these two models. Table 9 reports the highest equal-fold mean absolute SHAP values.

Table 9. Top mean absolute SHAP features averaged equally across outer folds.

Fold-specific global top-20 feature sets varied substantially across outer folds. Mean and median pairwise Jaccard similarity were 0.0597 and 0.0526 for the all-feature female model and 0.0442 and 0.0256 for the visual-feature male model. The fold-stability results are reported in Supplementary Section S1. Feature-name abbreviations are defined in Supplementary Section S5 (Supplementary Table S12).

To examine how feature importance varied over the conversation, mean absolute SHAP values were aggregated by 1-min interval, normalized within each outer fold, and averaged with equal fold weights. The importance shares ranged from 8.50% to 11.89% for the all-feature female model and from 8.73% to 13.33% for the visual-feature male model. The largest shares occurred at 6–7 min for the female model (11.89%) and at 7–8 min for the male model (13.33%). Table 10 reports the importance shares for all 10 intervals. The 95% bootstrap intervals summarizing variation across folds are provided in Supplementary Section S2.

Table 10. Fold-normalized mean absolute SHAP shares by 1-min interval.

Values sum to 100% within each model. Bold values indicate the largest proportions for the male and female models, respectively.

Among 3,156 comparable pairs of the same features in adjacent intervals, the median absolute correlation was 0.493; 49.2% had |r|≥0.5 and 28.4% had |r|≥0.7. These values indicate that features from adjacent 1-min intervals often contained overlapping information.

5 Discussion

5.1 Prediction performance of the machine-learning models

The highest observed weighted F1 was obtained with the all-feature configuration for the female model and the visual configuration for the male model. In the comparison restricted to the 613 dialogues shared by all female-model configurations, every 95% interval for the difference between the all-feature and each unimodal configuration included zero; thus, a clear advantage over unimodal modeling was not established. The numerical ordering is compatible with a potential benefit of multimodal integration, but the present comparison does not establish a reliable performance improvement. This possibility is consistent with evidence that multimodal performance can vary with the combination of modalities and the fusion strategy (Baltrušaitis et al., 2019; Ramachandram and Taylor, 2017; Gunes and Piccardi, 2005).

The class-specific metrics showed different prediction patterns in the all-feature female model and visual-feature male model. For the all-feature female model, F1 was 0.7647 for Not Exchange and 0.2987 for Exchange, with Exchange recall of 0.2500. Its weighted F1 therefore largely reflected performance for Not Exchange. By contrast, the visual-feature male model had class-specific F1 values of 0.5336 and 0.5425, indicating similar performance for the two classes. Visual behavior has also been used for contact-exchange prediction in existing speed-dating research (Veenstra and Hung, 2011). In the present study, the visual-feature configuration also produced the highest weighted F1 for the male model, suggesting that visual behavior may be relevant to predicting male participants' contact-exchange decisions.

Random Forest weighted F1 was numerically higher than the mean class-proportion random baseline in all eight configurations. Because every 95% interval for the difference included zero, a clear advantage over random prediction was not established. Nevertheless, the common numerical direction across all configurations motivates further evaluation of whether behavioral features carry predictive information in larger datasets.

5.2 Feature-importance patterns

Mean absolute SHAP analysis highlighted facial-movement features in both the all-feature female model and the visual-feature male model. In particular, AU04-related features from female participants ranked highly in both the all-feature female model and the visual-feature male model. AU04 represents brow lowering estimated by OpenFace, and this result suggests that the observed movement may be informative for predicting contact-exchange decisions. AU07- and AU12-related features also ranked highly in the visual-feature male model. This suggests that the model drew on multiple visual cues rather than on a single facial movement.

The rankings and compositions of the top-20 feature sets varied substantially across outer folds. However, AU04-related features had high aggregate importance in both the all-feature female model and the visual-feature male model. This cross-model pattern lends some support to the importance of AU04-related facial movement. However, the rankings of individual features defined by specific time intervals and statistics require confirmation in larger independent datasets.

As shown in Table 10, feature importance was distributed across the conversation rather than concentrated in a single 1-min interval. The largest shares were 11.89% at 6–7 min for the all-feature female model and 13.33% at 7–8 min for the visual-feature male model. These periods were therefore relatively important, but they did not account for the predictions on their own. The same features in adjacent 1-min intervals also contained similar information. Together, these results suggest the potential value of examining behavior and its temporal progression across the full conversation rather than focusing only on the interval with the highest importance.

5.3 Limitations

This study has several limitations that should be considered when interpreting its findings.

First, the scope of the findings is limited by the corpus. This study examined 10-min speed-dating interactions among Japanese participants. Future studies should test whether similar patterns are observed across different cultures, languages, and interaction durations.

Second, limitations remain regarding the dataset size and model comparison. The all-feature configuration used 3,650 predictors for 613 dialogues and was evaluated using participant-level nested cross-validation. However, because the number of predictors greatly exceeded the number of dialogues, overfitting remains possible. The classifier comparison was also limited to Random Forest and one L2 logistic-regression specification applied to the two main configurations. Model performance and feature importance should therefore be evaluated using larger independent datasets and a broader range of classifiers.

Third, the performance intervals capture only part of the uncertainty in model evaluation. The 95% intervals for performance and differences from the random baseline were calculated from the saved test predictions. They reflect variation among test participants but not variation that could arise from repeating model training, tuning, and feature selection. Repeating the full analysis procedure would provide a more comprehensive assessment of performance uncertainty.

Fourth, the comparison of female and male participants' decisions is limited by the use of separate models. The observed differences are therefore not a formal test of gender differences. A direct comparison would require a single model that tests category-by-feature relationships while accounting for repeated observations from participants.

Fifth, the feature-importance results have interpretive limitations. SHAP analysis describes which features the fitted models used for prediction but does not show that a behavior caused a contact-exchange decision. Mean absolute SHAP describes the magnitude of importance but not whether a feature was associated with predictions toward Exchange or Not Exchange, and feature rankings varied across folds. The reported 95% intervals also reflect only variation among folds. These findings should therefore be interpreted as patterns of model reliance whose stability requires evaluation in larger independent datasets.

Sixth, the temporal analysis depends on the selected segmentation. This study divided each conversation into 1-min intervals and compared SHAP importance from models using all 10 intervals. However, we did not remove specific intervals, shuffle information among intervals, or compare models using progressively longer portions of the conversation. The interval-level SHAP shares therefore do not represent the predictive performance of each interval in isolation. Comparisons using different interval lengths and temporal windows are needed to evaluate the robustness of the temporal patterns.

Seventh, missing data and tracking quality may have affected the results. This study used complete-case analysis and calculated dyadic OpenFace correlation features without thresholds based on tracking-success or confidence flags. If missingness or tracking quality was related to participant behavior or contact-exchange decisions, the estimates may have been affected. Comparisons using alternative missing-data methods, tracking-quality thresholds, and outlier procedures are needed to assess robustness.

Finally, the scope of the interaction features is limited. The current interaction features capture broad coordination patterns rather than the full range of dyadic behavior. Adding turn-taking, response latency, overlap, backchannels, and gaze coordination would enable more detailed modeling of interaction during the conversation.

6 Conclusion

This study used time-segmented multimodal behavioral features to predict contact-exchange decisions separately for female and male participants. The best-performing feature configuration differed between the two decision groups: the all-feature configuration performed best for female participants' decisions (weighted F1 = 0.6248), whereas the visual configuration performed best for male participants' decisions (weighted F1 = 0.5379). Across all evaluated feature configurations, the Random Forest models achieved numerically higher weighted F1 scores than random prediction; however, the uncertainty intervals for these differences included zero. At the class level, the all-feature model for female participants' decisions predicted Not Exchange more effectively than Exchange, whereas the visual model for male participants' decisions produced similar F1 scores for the two classes.

Mean absolute SHAP analysis identified the behavioral features and time intervals on which each model relied most strongly. AU04-related features ranked among the most important aggregate features in both best-performing models, although their rankings varied across folds. Temporal importance was distributed across all 10 intervals and was highest at 6–7 min for female participants' decisions and at 7–8 min for male participants' decisions. Overall, these findings suggest that temporally organized facial behavior may be informative for predicting contact-exchange decisions, although further evaluation using independent data is needed to establish the stability and generalizability of the observed patterns.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI tools (ChatGPT, OpenAI, Gemini, Google) were used to translate the manuscript from Japanese into English and to proofread and refine the English text. All scientific content was prepared and verified by the author(s), and all AI-assisted text was reviewed and revised by the author(s).

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomp.2026.1893669/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Andrea Generosi, Pegaso University, Italy

Reviewed by: Abdullah Ayub Khan, Bahria University, Pakistan

Md.Hamid Hosen, Elite Research, LLC, United States