Facial attribute-aware DeepFake detection through semi-supervised facial attribute labeling

1 Introduction

DeepFake technologies have rapidly evolved in recent years, increasing concerns for digital security, privacy, and public trust. These synthetic media techniques can generate highly realistic manipulated videos that are often difficult to distinguish from authentic content, especially when evaluated under heterogeneous manipulation methods and challenging compression settings (Tolosana et al., 2020; Verdoliva, 2020; Li et al., 2020; Rössler et al., 2019). Although substantial progress has been made in DeepFake detection, current systems still face important limitations in cross-condition robustness and generalization. Traditional approaches based on fixed artifact patterns, frequency-domain traces, or facial inconsistency cues may achieve strong performance on specific benchmarks, yet they often degrade when the visual characteristics of the data change or when forgeries become better blended and more realistic (Durall et al., 2020; Li et al., 2018; Yang et al., 2019).

Recent research has increasingly explored interpretable signals and semantic clues to improve the transparency and reliability of DeepFake detection systems. Prior studies have investigated facial dynamics, head-pose inconsistencies, attention-based temporal modeling, and multimodal or artifact-aware strategies as ways to improve forensic analysis (Guera and Delp, 2018; Li et al., 2018; Yang et al., 2019; Dang et al., 2020; Neekhara et al., 2021). However, most of this literature remains primarily oriented toward architectural improvements or handcrafted forensic cues, while the role of high-level facial attributes in shaping prediction behavior remains comparatively underexplored. In particular, automatic facial attribute labeling has rarely been used as a structured instrument to analyze error patterns, diagnose potential subgroup disparities, and study whether some visible characteristics systematically facilitate or hinder DeepFake detection.

To address this gap, we first enrich the FaceForensics++ benchmark with high-level facial attributes obtained through a semi-supervised labeling pipeline. Starting from a manually annotated seed subset, we propagate labels to the remaining videos in order to obtain attribute information for characteristics such as gender, hair color, hair length, ethnicity, and ear visibility. This first stage of the study provides the methodological foundation for an attribute-aware analysis of detector behavior, see Figure 1 (Stile et al., 2026). Rather than using these attributes as an end in themselves, we use them to examine whether misclassifications are unevenly distributed across visually meaningful categories and to determine whether these categories can reveal systematic weaknesses in the detector (Rössler et al., 2019; Guerrero-Contreras et al., 2024, 2025).

The preliminary evidence obtained from this attribute-aware error analysis motivates a second and more controlled experimental stage. Instead of limiting the investigation to descriptive correlations between labels and wrong predictions, we examine whether the absence of specific attribute values during training affects the detector's ability to generalize to a complete and fixed test distribution. To this end, we conduct controlled training-time exclusion experiments in which samples associated with selected facial attributes are removed from the training set, while all resulting models are evaluated on the same unified test set. This protocol makes it possible to isolate subgroup-conditioned generalization effects and to determine whether the disparities observed in the initial analysis correspond to genuine robustness limitations rather than incidental fluctuations in descriptive statistics.

Within this unified framework, the present study combines the methodological contribution of semi-supervised facial attribute labeling with an attribute-aware training analysis centered on controlled exclusions, see Figure 2. The paper shows that facial attributes are not only useful for post-hoc interpretability, but also informative for stress-testing detector robustness under distribution shifts. In particular, the results identify ear visibility as the most critical factor for reliable FAKE vs. REAL discrimination, while hair length exhibits a moderate but consistent effect that interacts with ear visibility. These findings support the adoption of bias-aware training strategies based on targeted data curation, attribute-specific augmentation, and threshold calibration, and they contribute to the development of more interpretable and operationally reliable DeepFake detection systems.

2 Materials and methods

2.1 Dataset and study design

The proposed methodology begins with the definition of a specific dataset containing REAL and DeepFake videos. Frames are then extracted, successively processed, and split into training, validation, and test sets in order to train a neural network able to carry out a frame-level classification task to distinguish pristine from fake contents. In parallel, the data used for evaluation are labeled according to a set of facial attributes through a semi-supervised procedure. Based on the output of the labeling process, we investigate whether the predictions produced by the trained neural network exhibit systematic errors correlated with specific high-level facial attributes. Such possible relations are then extended through controlled training-time exclusions on a unified test set to analyze generalization bias.

We utilize FaceForensics++, a benchmark dataset widely adopted in DeepFake research (Rössler et al., 2019). The original corpus consists of 1,000 pristine YouTube videos, manually screened to ensure near-frontal, occlusion-free faces at a resolution of at least 480p, which serve as targets for automated manipulations. The manipulated portion of the dataset is organized into six families: Deepfakes, FaceSwap, Face2Face, NeuralTextures, FaceShifter, and DeepFakeDetection, see Table 1. Each family comprises 1,000 videos generated from the same 1,000 sources, for a total of 6,000 manipulated and 1,000 real videos. Per-frame ground-truth masks of the edited regions are also available for localization research (Rössler et al., 2019).

Table 1. FaceForensics++ contents with manipulation families.

The bold values indicate the total number of videos in the dataset, comprising 1,000 original pristine videos and 6,000 deepfake forged videos, yielding a cumulative total of 7,000 videos.

For the purposes of this study, we selected the 1,000 REAL videos from the /youtube folder and their corresponding 1,000 FAKE videos from the /Deepfakes folder, created with the faceswap software.¹ We focused our analysis on this specific forgery method and used only the C40 compression level throughout the experiments, see Table 2 for more details. All reported results are therefore based on this compression configuration.

Table 2. FaceForensics++ video encodings and compression settings.

Training practices were standardized from the outset. Images were normalized, and the dataset was split into 72% training, 14% validation, and 14% test sets, using stratification across REAL and FAKE classes to preserve class balance during model selection and final reporting. This protocol provided a stable baseline for subsequent analyses and comparisons. Building on this foundation, the work proceeded along two connected directions: first, completing semi-supervised labeling on FaceForensics++ and using those labels to analyze misclassification patterns, and second, refining this analysis through controlled exclusion experiments on a shared evaluation protocol.

2.2 Face extraction and pre-processing

The objective of this preprocessing step was to extract face-only image frames from the FaceForensics++ dataset. Face detection was performed using the classical Haar Cascades method. The OpenCV detectMultiScale function was applied on grayscale versions of each frame. The scaleFactor parameter, set to 1.3, controls the image pyramid reduction ratio during face detection, enabling the model to detect faces at different scales. The minNeighbors parameter, set to 5, defines the minimum number of adjacent rectangles required to retain a detection, acting as a threshold for eliminating false positives. These values were selected empirically, balancing detection accuracy and noise reduction.

Detected faces were cropped, resized to 224 × 224 pixels, and saved as individual .jpeg files, see Figure 3. While effective, this pipeline presents notable challenges in terms of data volume, introducing complexity in both storage and computational overhead during training. To mitigate this, we adopted a frame-skipping strategy with a fixed sampling rate of skip = 20, meaning one frame was retained for every twenty. This choice preserves a sufficient level of temporal representation of the source videos and reduces redundancy, helping prevent overfitting and model bias from near-duplicate frames.

The final dataset used in the initial frame-level detection study consisted of 28,483 REAL facial frames extracted from the 1,000 videos in the youtube folder and 28,051 FAKE facial frames extracted from the 1,000 videos in the Deepfakes folder. From this collection, three subsets were constructed for training, validation, and testing. The training set contains 40,320 facial images, equally divided between FAKE and REAL instances. The validation set consists of 7,840 images, again balanced between classes, and the test set includes 7,840 images with an equal number of REAL and FAKE faces.

2.3 Semi-supervised facial attribute labeling

The CSV metadata distributed with FaceForensics++ provide only technical descriptors, namely Index, File path, Label (REAL or FAKE), Frame count, Width, Height, Codec, and File size (MB). These fields are operationally useful and were used to drive preprocessing choices in the pipeline, for example decoding, frame sampling, and compression stratification. They do not, however, encode qualitative or semantic information about the visual content. As a consequence, they do not support analyses that are central to interpretable DeepFake research, such as attribute-conditioned performance reporting, bias diagnosis, or failure-mode inspection tied to concrete visual traits. This motivated a targeted facial attribute labeling phase.

To annotate the dataset, we adopted a semi-supervised labeling approach (Guerrero-Contreras et al., 2025). As a first step, a ground truth was established by manually inspecting the first 50 REAL videos from the /youtube folder of FaceForensics++, specifically 000.mp4 through 049.mp4. Each video was reviewed frame by frame to assign a comprehensive set of high-level visual and semantic labels.

The annotation covered categorical features such as gender (FEMALE, MALE), ethnicity (AFRICAN, ASIAN, MIXED, WHITE, OTHER), expression (ANGRY, DISGUSTED, HAPPY, LAUGHING, SERIOUS, SURPRISED), hair_color (BLONDE, BLACK, BROWN, GREY, LIGHT BROWN, RED, OTHER), hair_length (BALD, LONG, PONYTAIL, SHORT), action (HUGGING, STILL, WALKING), and position (SITTING, STANDING, UNKNOWN). In parallel, boolean attributes were collected, including whether the subject is alone, whether ears or forehead are visible, whether glasses are worn, and whether the video exhibits camera movement, pan effects, shakiness, or active talking, each recorded as TRUE or FALSE. All labels were assigned based on traits that are visible and consistent throughout the clip.

For this study, each manipulated video generated from a given real video via the DeepFakes method, and stored in /Deepfakes, was assumed to inherit the same facial attribute labels as its source. This assumption is plausible because DeepFakes primarily performs an identity swap while preserving most coarse facial characteristics of the target subject, including stable traits such as apparent gender, and in practice the manipulation does not systematically alter these attributes across the dataset. Gender, the characteristic that could raise the greatest uncertainty, proved to be highly stable in this setting. All 50 manually annotated real videos preserved the same gender presentation in their corresponding DeepFake counterparts, and additional spot checks on randomly selected video pairs across the dataset did not reveal any counterexamples. Based on this evidence, the same transfer assumption was extended to the full /Deepfakes subset.

The output of this phase was the creation of a CSV file named attribute_youtube.csv, containing the manually assigned labels for the seed subset.² An example of the annotation structure is provided in Supplementary material.

Using 5% of the /youtube videos, corresponding to 50 clips, as human-labeled seeds, a self-labeling procedure was then applied to produce pseudolabels for the remaining 95%, corresponding to 950 clips. The auto-labeling models use a ResNet18 backbone and are trained separately for each attribute as binary or multiclass classifiers depending on the label type (Guerrero-Contreras et al., 2024, 2025). Training follows an iterative loop. First, the 50 seeds are split 60-20-20 into training, validation, and test sets, corresponding to 30, 10, and 10 videos. A pretrained ResNet18 is fine-tuned on the training portion with early stopping on validation. The resulting model is evaluated on the test split to establish a baseline, then applied to pseudolabel the 950 unlabeled clips. Only samples with confidence above 95% are admitted into the augmented training set, which is then used for further fine-tuning. The augmentation and retraining loop repeats until convergence on the held-out test set or until no additional high-confidence samples remain.

From the initial pool of attributes, features such as gender, hair_color, and is_ears_visible showed higher predictive utility and were retained. Attributes with a single observed class in the seed set were excluded to avoid severe class imbalance. The final set of selected attributes used in the analysis is reported in Table 3.

Table 3. Selected facial attributes used for analysis.

2.4 Evaluation metrics

To rigorously assess the performance of the DeepFake detection model, a comprehensive set of evaluation metrics was employed. During model training, accuracy and loss metrics were tracked over each epoch for both the training and validation sets in order to detect signs of overfitting or underfitting. Predictions on the test set were generated using a sigmoid output activation with a classification threshold of 0.5.

The following binary classification metrics were computed: accuracy, precision, recall, F1-score, and confusion matrix. Accuracy measures the overall percentage of correctly classified frames. Precision is defined as the proportion of predicted FAKE frames that are truly FAKE. Recall measures the proportion of truly FAKE frames the model is able to detect. The F1-score is the harmonic mean of precision and recall, and the confusion matrix summarizes the counts of true positives, false positives, true negatives, and false negatives.

To gain a deeper understanding of model behavior, frame-level statistics were also considered. In particular, we computed and compared the number of frames predicted as FAKE or REAL, calculated how many of those predictions were correct, and identified false positives and false negatives. Misclassified frames were then cross-referenced with the automatically labeled facial attributes in order to investigate whether certain visual characteristics were overrepresented in erroneous predictions.

To explore potential biases in the DeepFake detection pipeline, three statistical correlation metrics were computed between the presence of misclassifications and each attribute label: the chi-square test statistic, the associated p-value, and mutual information. Given observed and expected frequencies, the chi-square statistic is defined as Equation 1:

χ2=∑i∑j(Oij-Eij)2Eij(1)

The resulting p-value estimates the probability that a discrepancy at least as large could arise under the null hypothesis of independence. The conventional decision rule adopts the threshold p < 0.05 to claim statistical significance (Fisher, 1925). Complementarily, mutual information quantifies how much information the attribute label X conveys about the correctness indicator Y (Equation 2):

MI(X,Y)=∑x∑yp(x,y)log(p(x,y)p(x)p(y))(2)

In the controlled exclusion experiments, performance was reported globally and by subgroup using accuracy, F1, AUC, and confusion-derived rates. In particular, the true positive rate and true negative rate were used to summarize sensitivity to REAL and FAKE decisions under a shared evaluation protocol. Furthermore, a minimum subgroup size threshold was applied during subgroup analysis: any attribute subgroup with fewer than five instances was excluded from the plots and summaries in order to avoid unstable estimates driven by very small sample sizes.

2.5 Attribute-aware analysis and controlled exclusion protocol

The first stage of the study used the labeled attributes to conduct a structured analysis of misclassification patterns. The second stage extended this direction through controlled exclusion experiments designed to assess subgroup-conditioned generalization gaps. Building on the frame-level detection and attribute-labeling pipeline already described, this phase adopts FaceForensics++ as benchmark and restricts the analysis to the /youtube REAL subset and its corresponding /Deepfakes FAKE subset under the C40 compression level. Videos are decoded and uniformly sampled to extract frames, faces are detected and cropped with the same preprocessing pipeline, and high-level facial attributes are obtained through the semi-supervised procedure described above.

For each attribute A and value v∈VA, the protocol follows two steps: first, exclusion training, where all samples with A = v are removed from the training set; second, universal testing, where the resulting model is evaluated on the complete test set that includes all values of A, including v. This design isolates the generalization bias induced by the systematic absence of a subgroup during learning while keeping the test distribution fixed.

In the latest runs, the training set consisted of N = 100, 000 balanced facial frames, and the classifier was trained for up to 5 epochs, with model selection on validation performance. All runs were carried out with a fixed random seed and compared against a no-exclusion baseline model. This controlled protocol makes it possible to determine whether the attribute-dependent disparities observed in the initial descriptive analysis correspond to genuine robustness limitations under distribution shift.

3 Results

↓ R2: The section below is moved from Supplementary material.

3.1 DeepFake detection pipeline

Although various convolutional neural networks were evaluated during the frame-level DeepFake detection phase, including architectures such as ResNet50, the results reported here are based on a VGG16-based model. Performance across different architectures was highly comparable in terms of accuracy, precision, and F1-score, while VGG16, initialized with ImageNet pre-trained weights and with all layers frozen during training, demonstrated reliable and consistent behavior, particularly on C40-compressed facial images and due to its ease of deployment on standard hardware.

The classifier was based on the VGG16 architecture with pre-trained ImageNet weights and a simple custom head composed of a Flatten layer followed by Dense(512, relu) and a final Dense(1, sigmoid) output. Training was performed with the Adam optimizer, a learning rate of 0.0001, binary cross-entropy loss, a batch size of 16, and 5 epochs.

The test set was used to evaluate the final model. Before evaluation, a face-filtering step was applied so that, for each video, only one face track was retained, corresponding to the main subject. In the implementation, this primary track is denoted as face0, and it is treated as the candidate manipulated or authentic face for that clip.³

Although classification is performed at the frame level, video-level decisions are also considered, since system reliability in DeepFake detection is typically evaluated per video rather than per frame. Following the protocol adopted in the study, frame-level predictions can therefore be aggregated to obtain a more robust video-level decision.

3.2 Baseline DeepFake detection performance

The experimental evaluation first assessed the effectiveness of the DeepFake detection pipeline based on visual features extracted from facial frames. The model was trained on a balanced dataset of 40,320 facial images, with 20,160 REAL and 20,160 FAKE samples, using a validation set of 7,840 images equally split between classes and a test set of 7,840 images also balanced across the two classes, as reported in Table 4. The classifier was based on the VGG16 architecture with pre-trained ImageNet weights and frozen convolutional layers, followed by a simple dense head. After training, the model reached a training accuracy of 94.58% and a validation accuracy of 86.96%.

Table 4. Dataset split by set and class.

The test set was then used to evaluate the final model. Before evaluation, a face-filtering step was applied so that, for each video, only one face track was retained, corresponding to the main subject. In the implementation, this primary track is denoted as face0, and it is treated as the candidate manipulated or authentic face for that clip. The resulting frame-level classification performance is summarized in Table 5, while the confusion matrix is shown in Figure 4. On the frame-level test set restricted to face0, the model achieved an accuracy of 85.26% across 6,967 frames.

Table 5. Frame-level classification report on the face0-only test set.

Although classification is performed at the frame level, system reliability in DeepFake detection is typically evaluated at the video level. According to established literature, it is not methodologically sound to mark an entire video as misclassified based on a single erroneous frame. Rössler et al. (2019); Sabir et al. (2019) suggest aggregation strategies such as mean score, majority voting, or top-K frame voting to derive a more robust video-level decision. In the protocol adopted here, a video is considered misclassified only if it contains at least three wrongly predicted frames. This threshold is particularly appropriate in this case, as each video in the test set contains an average of 28 extracted frames.

As a direct consequence, a substantial portion of the dataset composed of 284 videos with only one misclassified frame and 115 videos with exactly two misclassified frames was not marked as erroneous in the video-level analysis. To better reflect real-world deployment conditions, where decisions are typically made at the video level rather than per frame, predictions were aggregated by assigning each video the label associated with its corresponding face0 frame. After defining the criteria for classifying a video as mispredicted and analyzing several edge cases, a total of 128 videos were identified as misclassified at the video level. Within the adopted evaluation protocol, the detector misclassified 123 genuinely FAKE videos as REAL and five genuinely REAL videos as FAKE.

When tested at this video granularity, the system achieved a video-level test accuracy of 93.25%. The resulting confusion matrix is shown in Figure 5, while a detailed classification report for video-level predictions is reported in Table 6.

Table 6. Video-level classification report.

↑ R2: The section above is moved from Supplementary material.

Before the attribute-aware analyses, we established the baseline performance of the DeepFake detector on the selected FaceForensics++ subset. Using the face0-based evaluation protocol, the detector achieved an accuracy of 85.26% at the frame level on 6,967 frames and 88.40% at the video level. These baseline results provided the reference point for the subsequent attribute-aware error analysis and controlled exclusion experiments. Detailed architectural settings and additional frame-level and video-level reports are provided in Supplementary material.

3.3 Initial attribute-aware error analysis

R3: To investigate whether the detector exhibits biases linked to specific facial attributes, a quantitative analysis was conducted by correlating classification errors with high-level attribute labels. In particular the overall Accuracy, which serves as the de-facto standard benchmark in DeepFake detection literature, were computed for each category within five features: gender, ethnicity, hair color, hair length, and ear visibility. These metrics provide complementary insights. Precision indicates the proportion of correct predictions among all predictions for a given label and the error rate reflects the proportion of misclassified instances out of the total samples for each label. As a starting point for our observations, global performance metrics across different facial attribute subsets are reported in Table 7.

Table 7. DeepFake detection accuracy across different facial attribute subsets.

The test set comprises a unified group of 2,000 videos (1,000 authentic and 1,000 manipulated) to ensure statistical consistency across all sub-group analyses.

R3: While these aggregated metrics suggest uniform behavior, they often obscure local failure modes. To uncover these hidden patterns, we analyzed the video-level confusion matrix shown in the Table 8 and conducted a rigorous label-wise statistical correlation analysis in the Table 9.

Table 8. Model architecture and training configuration.

Table 9. Label-wise classification metrics computed over the test video set.

From this analysis few patterns clearly emerged, individuals labeled with LONG hair showed the highest accuracy score, equal to 0.763, compared with SHORT hair, whose accuracy score was 0.700. Conversely, videos featuring subjects with SHORT hair presented the highest error rate, equal to 0.300, indicating a potential challenge for the model in processing these cases. Gender-wise, the model showed slightly better performance on FEMALE than on MALE, although the difference remained marginal. Labels with very few samples, such as BALD, PONYTAIL, or OTHER for hair color, displayed perfect or near-perfect scores, but these values cannot be interpreted as robust evidence because the sample sizes are too small to support statistically reliable conclusions.

To further explore potential biases in the pipeline, three statistical correlation metrics were computed between the presence of misclassifications and each attribute label: the chi-square statistic, the associated p-value, and mutual information. The results are summarized in Table 10. Notably, the attribute hair_length stands out with a statistically significant p-value of 0.031793, suggesting that the likelihood of a wrong prediction is not evenly distributed across different hair-length categories. This result is consistent with the descriptive metrics reported above, where SHORT hair length was associated with the highest error rate.

Table 10. Statistical correlation metrics between features and wrong predictions.

R#3: Expanding upon this analysis by measuring the association between specific attribute labels and classification errors using the Chi-Square (χ²) test of independence, corresponding asymptotic p-values, and Mutual Information (MI), we mapped the systematic vulnerabilities of the detector. These statistics are summarized in Table 11. The label hair_length = SHORT exhibited a statistically significant correlation with misclassifications, with p < 0.05, suggesting a potential association between this attribute and increased prediction errors. While ethnicity = WHITE also showed statistical significance, this finding should be interpreted with caution because this label is overwhelmingly dominant in the dataset, and the higher error count may primarily reflect prevalence rather than a specific model weakness toward this subgroup. By contrast, hair_length = LONG, although slightly above the threshold for statistical significance, also showed the highest F1-score among the hair-length categories.

Table 11. Label-wise statistical correlation metrics with wrong predictions.

In addition to the literature-based thresholding described above, the distribution of wrongly predicted frames per video was further examined, and the subset of 13 videos with more than five misclassified frames was manually inspected through visual analysis. This inspection identified three misleading cases, namely 548_632, 305_513, and 554_572, in which the detected face0 did not correspond to the actual manipulated subject, but rather to a background face, a non-target subject, or a shadow-like pattern. By contrast, video 186_170 contained additional background faces but did not introduce a spurious target mismatch. To ensure dataset integrity, the three misleading videos were excluded from the subsequent analysis, and videos with fewer than three wrongly predicted frames were discarded according to the adopted video-level thresholding strategy. Additional qualitative examples of anomalous videos and the distributions of wrong predictions before and after filtering are reported in Supplementary material.

The attribute-aware error analysis therefore provided a first indication that some appearance-related factors, particularly hair_length, and to a lesser extent is_ears_visible, were associated with non-uniform error distributions. These preliminary findings motivated a second and more controlled experimental stage.

3.4 Controlled exclusion experiments

The second stage of the study moved from descriptive correlation analysis to controlled training-time exclusions on a fixed and unified test set. For the no-exclusion model, denoted as excl-none, the detector obtained an accuracy of 0.806 and an AUC of 0.823. The corresponding confusion matrix revealed a skew toward the REAL class, with TN = 4, 715, FP = 2, 285, FN = 427, and TP = 6, 573. This yielded a true positive rate of 0.939 and a true negative rate of 0.674. These values are summarized in Table 12, while the corresponding confusion matrix is shown in Figure 6A.

Table 12. Performance metrics for the excl-none model.

Controlled exclusions on ear visibility were then performed under the same protocol. Compared with the no-exclusion baseline, excluding samples with is_ears_visible = 1 yielded the largest degradation, with an accuracy of 0.741 and an AUC of 0.763. By contrast, excluding samples with is_ears_visible = 0 had a much milder effect, yielding an accuracy of 0.813 and an AUC of 0.832. These results are reported in Table 13.

Table 13. Controlled exclusion on ear visibility, VGG16-based classifier, unified test set.

The confusion matrices corresponding to the two exclusion conditions are shown in Figures 6B, C. The model trained excluding samples with is_ears_visible = 0 displayed a performance comparable to the baseline, with slightly higher specificity and slightly lower sensitivity. By contrast, the model trained excluding samples with is_ears_visible = 1 represented the worst case, with a marked drop in both sensitivity and specificity.

These controlled exclusion experiments showed that the attribute-dependent disparities suggested by the initial descriptive analysis corresponded to genuine robustness limitations under distribution shift rather than to incidental fluctuations in descriptive statistics.

3.5 Focus on hair length

Among the analyzed attributes, hair_length emerged in the initial study as the most relevant factor in the statistical correlation analysis. In the test set, SHORT and LONG were widely represented, while PONYTAIL remained rare and its metrics should therefore be interpreted with caution because of the limited number of samples.

Without exclusions, performance was slightly higher on LONG than on SHORT, with PONYTAIL showing optimistic figures that reflect the low sample count. Excluding SHORT during training produced a noticeable decline when generalizing to SHORT at test time, consistent with a generalization gap. Excluding LONG was more tolerable, improving robustness on SHORT with a manageable penalty on LONG. Overall, hair_length behaved as a context feature: it mattered and had a moderate but consistent effect, yet it did not represent the most critical factor in the system.

These results are consistent with the initial attribute-aware analysis, where SHORT hair showed both the highest error rate and a statistically significant correlation with wrong predictions. The controlled exclusion experiments therefore confirmed that hair_length has a moderate but recurrent influence on detector behavior.

3.6 Focus on ear visibility

R4: A critical methodological finding of this study is the dramatic divergence in behavior observed between the initial descriptive analysis and the subsequent causal exclusion experiments. In the baseline model trained on the full dataset, the critical importance of the is_ears_visible attribute was largely masked, showing an illusion of uniform performance across macro-categories. The vulnerability only crystallizes when transitioning from passive post-hoc auditing to an active training with causal exclusions. The severe degradation observed when excluding visible ears (Accuracy dropping to 0.741) vs. the high stability maintained when excluding non-visible ears (0.813) uncovers a directional structural dependency. This directly proves that while traditional macro-averaging irons out localized drops under full-data regimes, targeted exclusion testing is mathematically required to expose the brittle structural dependencies and hidden blind spots embedded within the deepfake detector.

Ear visibility represented the most critical factor in the controlled exclusion study. This attribute changes the visual geometry of the face through lateral occlusions, hairline shape, peri-auricular contours, and the distribution of skin vs. hair textures. In the no-exclusion setting, the detector showed similar performance on is_ears_visible = 1 and is_ears_visible = 0, with a small advantage for visible ears, suggesting a latent dependency on this detail.

When training excluded samples with is_ears_visible = 1, the detector exhibited the worst performance observed in the experiments, with an accuracy of 0.741 and an AUC of 0.763. The model became biased toward REAL and allowed many FAKE samples to pass, indicating a collapse in specificity when the experience of visible ears was removed during learning. By contrast, when training excluded samples with is_ears_visible = 0, performance did not degrade and was slightly higher than the baseline, with an accuracy of 0.813 and an AUC of 0.832. This suggests that cases with non-visible ears provide relatively weak or noisy supervisory signal for this detector, so removing them from training does not harm, and may even stabilize, the decision boundary.

For readability, the legend of the experimental runs is included in the caption of Figure 7. The global degradation pattern is visible across subgroup plots for gender, hair color, hair length, and ear visibility itself.

Taken together, both exclusion experiments converged on the same point: is_ears_visible is the highest-impact attribute in the proposed framework. Removing is_ears_visible=1 from training produced a marked drop in both accuracy and AUC, together with an unsafe increase of FAKE samples classified as REAL. Removing is_ears_visible=0 had a non-impacting effect and could even improve performance. In operational terms, coverage of ear visibility conditions is therefore a critical requirement for robust detection.

3.7 Global interpretation of confusion asymmetry

The confusion matrices observed in the controlled exclusion experiments revealed a recurrent asymmetry, particularly in the form of FAKE samples classified as REAL. This behavior is consistent with the baseline operating point. For the no-exclusion model, the AUC indicates good ranking ability, yet the fixed decision threshold of 0.5 favors recall on REAL at the expense of specificity on FAKE. In practice, threshold calibration on the validation set, for example selecting the threshold that maximizes Youden's J or equalizes true positive rate and true negative rate, would shift the operating point along the ROC curve and reduce this asymmetry (Verdoliva, 2020; Rana et al., 2022).

A second source of asymmetry arises at the video level, where frame scores are aggregated with a rule such as classifying a video as FAKE if at least k frames are predicted as FAKE. With k = 3, a detector can still miss a FAKE video when manipulations are temporally sparse or when face detection fails on a fraction of frames, while a REAL video rarely accumulates three false positives. This mechanism tilts errors toward the FAKE → REAL direction. This behavior is consistent with prior observations that thresholding and protocol choices substantially affect reported performance in FaceForensics++ and related benchmarks (Rössler et al., 2019; Li et al., 2020). Moreover, high-quality or well-blended fakes can compress artifacts and reduce separability, increasing FAKE to REAL errors at conservative thresholds (Beckmann et al., 2023).

The overall results therefore reveal three recurring patterns. First, there is an overall degradation when a decision-relevant subgroup, such as is_ears_visible = TRUE, is excluded from training. Second, a rebalancing between true positive and true negative rates occurs, with the model becoming either more conservative or more permissive depending on the excluded subgroup. Third, the detector shows distribution sensitivity, with larger impacts when the excluded subgroup is numerically relevant or corresponds to a structural variation of facial appearance. Within this framework, two attributes emerge as particularly influential: hair_length, which shows a moderate yet consistent effect, and is_ears_visible, which proves markedly critical for correct FAKE vs. REAL discrimination.

3.8 Conclusion and future work

R6: In this study, we introduced a structured framework to investigate the intricate relationships between high-level facial attributes and the prediction vulnerabilities of DeepFake detectors. By leveraging a robust, human-in-the-loop semi-supervised labeling pipeline and conducting targeted training-time exclusion experiments, we demonstrated that global model performance metrics often mask localized subgroup dependencies—most notably identifying ear visibility as a critical geometric and biometric anchor for reliable real-vs.-fake discrimination.

A foundational pillar of our experimental design was the deliberate restriction of our evaluation to a single manipulation family within the FaceForensics++ corpus, called DeepFakes, specifically focusing on advanced identity swaps generated via deep learning-based architecture. From a scientific standpoint, prioritizing this single forgery type was methodologically necessary to eliminate confounding algorithmic variables and suppress software-specific artifact distributions.

However, this necessary methodological boundary opens significant opportunities for subsequent research, and we explicitly invite the scientific community to extend this framework toward a broader, cross-paradigm generalization. While the peripheral blending zones inherent to identity-swapping methods provide a universal justification for our findings regarding ear and hair perimeters, exploring how these dependencies behave across fundamentally different generative architectures represents a critical next step. Specifically, future trajectories should generalize this causal auditing workflow to encompass the entire taxonomy of DeepFake synthesis, including graphics-based reenactment (e.g., Face2Face) and neural rendering paradigms (e.g., NeuralTextures), where internal facial dynamics are modified while peripheral structures remain unchanged.

By scaling this attribute-aware stress-testing across diverse forgery classes and multi-attribute combinations, future efforts will be able to map the cross-condition robustness of biometric classifiers definitively. Ultimately, we hope that the insights and the validated dataset presented in this work will serve as a foundational benchmark, guiding the development of the next generation of bias-aware, interpretable, and operationally resilient deepfake detection systems.

4 Related works

R1: The methodological foundation of this study directly extends our preliminary conference work, which introduced the core paradigm of leveraging semi-supervised facial attribute recognition to dissect frame-level misclassifications in DeepFake detectors (Stile et al., 2026). While existing literature has recently moved toward attribute-aware annotations for FaceForensics++—most notably the Massively Annotated Deepfake Database (MADD) which provides a massive data resource for the community alongside rigorous, reproducible code and GitHub-distributed tools—our pipeline introduces two fundamental distinctions (Xu et al., 2024). First, whereas prior works like MADD rely almost entirely on fully automated estimation models to label thousands of videos, they remain inherently susceptible to secondary biases and systematic errors when processing manipulated or blurry deepfake regions. To ensure absolute label reliability, our framework implements a hybrid semi-supervised pipeline backed by strict manual auditing. Every automated attribute prediction in our system underwent human-in-the-loop verification and cross-referencing; this unprecedented level of ground-truth integrity is strictly mandatory for statistical bias audits, as noisy labels would inevitably corrupt the validity of the subsequent p-values. Second, while prior efforts position themselves primarily as passive benchmarking databases to test models across pre-existing categories, our framework goes beyond descriptive auditing. The verified, noise-free sub-attribute partitions produced here are explicitly designed to power training-time causal exclusion experiments, allowing us to actively isolate and mitigate hidden structural vulnerabilities under controlled distribution shifts.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI technologies were used during manuscript preparation for proofreading, language revision, translation, and text editing based on author-written drafts in the authors' native language or in UK English, as well as for converting images and tables into LaTeX according to publisher-specific formatting requirements. The tools used were ChatGPT Thinking 5.4 (OpenAI), Gemini 3.1 Pro (Google), and Mate Translate 8.3.3. These tools were not used to generate the study design, primary scientific claims, or final interpretation of the results. The authors reviewed and validated all AI-assisted outputs and take full responsibility for the accuracy, originality, citations, and final content of the manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fimag.2026.1846377/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Marco Arnesano, University of eCampus, Italy

Reviewed by: Seema Rawat, Amity University, India

Stelios Mylonas, Centre for Research and Technology Hellas (CERTH), Greece

Vishal Vora, Atmiya University, India

Footnotes

The deepfakes/faceswap project, created and maintained by Matt Tora (known by his pseudonym @torzdf), Andrey Ivanov (@andenixa), and Bryan Lyon (@bryanlyon), GitHub repository https://github.com/deepfakes/faceswap (visited on 2025-12-26).

File available on the public GitHub repository created by the author: https://github.com/vstile/deepfake-attribute-detection/blob/main/dataset/attribute/ground-truth/attribute_youtube.csv (visited on 2025-12-26).

When multiple faces are detected in a frame, the Haar Cascades detector stores them with incremental indices such as face0, face1, face2, and so on. In the FaceForensics++ YouTube videos considered here, there is usually a single central speaker, consequently face0 almost always corresponds to the most prominent face in the scene and is used as the reference face track for evaluation.