Reliability-aware knowledge distillation with self-evolving supervision for robust robotic visual perception

1 Introduction

Robotic autonomy in uncertain and dynamic environments relies heavily on reliable visual perception. Autonomous robots, such as mobile robots, unmanned vehicles, aerial platforms, and service robots, must interpret visual scenes under changing illumination, sensor noise, motion blur, partial occlusion, background clutter, and distribution shifts. Recent robotic perception systems, such as semantic SLAM methods that integrate visual and semantic cues (Cai et al., 2025), further demonstrate the importance of reliable scene understanding for autonomous operation. These factors make visual perception more challenging than standard image recognition under controlled conditions. In such scenarios, supervision signals may be noisy, biased, or partially incorrect, and models trained without explicitly considering supervision reliability may accumulate systematic errors and exhibit degraded generalization.

At the same time, robotic platforms are often constrained by computation, memory, energy consumption, and real-time response requirements. Although large-capacity visual models have achieved strong performance in classification, detection, and scene understanding, directly deploying them on resource-constrained robotic systems is often impractical. Knowledge distillation (KD) has therefore become an attractive solution for transferring knowledge from a high-capacity teacher model to a compact student model, enabling efficient visual perception while preserving discriminative performance.

However, conventional KD methods usually rely on the implicit assumption that teacher predictions are globally reliable. This assumption is problematic for robotic perception in uncertain environments. A teacher model may produce incorrect or biased predictions when encountering ambiguous visual patterns, rare categories, corrupted sensory inputs, or samples outside its competence boundary. If such predictions are directly transferred to the student, the student may inherit both useful knowledge and erroneous supervision, leading to negative transfer. This issue is particularly important for autonomous robots, where unreliable perception may further affect downstream decision-making, navigation, interaction, and task execution.

Most existing KD approaches treat the teacher as a dominant and trustworthy source of supervision, aiming to approximate the teacher as closely as possible. This strategy overlooks a fundamental limitation: teacher models inevitably contain errors and competence boundaries. When knowledge is transferred without distinguishing reliable and unreliable components, the student may not only learn useful patterns but also amplify the teacher’s mistakes. Therefore, a robust distillation framework for robotic visual perception should not simply compress the teacher, but should also identify, correct, and progressively refine unreliable supervision.

Motivated by the above observations, this work studies reliability-aware knowledge transfer for compact robotic visual perception and focuses on two key questions: (1) Which parts of the teacher’s knowledge are reliable enough to be transferred to a compact student model? (2) Can unreliable teacher supervision be corrected and progressively evolved so that the student becomes more resistant to negative transfer during knowledge distillation?

To address the first question, we revisit the role of the teacher in knowledge distillation from the perspective of reliability. Instead of treating teacher predictions as uniformly trustworthy, we explicitly distinguish reliable and unreliable knowledge by comparing teacher predictions with ground-truth labels. Reliable knowledge is directly inherited, while unreliable knowledge is identified and further refined. Based on this distinction, the student is encouraged to selectively learn from the teacher rather than blindly mimicking its outputs.

However, simply filtering unreliable knowledge is insufficient. Even when only reliable knowledge is transferred, the student may still be constrained by the teacher’s performance. To overcome this limitation, we further introduce a dynamic knowledge evolution mechanism that enables progressive refinement during training. In this framework, reliable knowledge serves as a stable foundation, while unreliable knowledge drives adaptive correction and refinement. Through iterative learning, the student gradually improves its predictions and reduces dependence on the teacher’s limitations.

The main contributions of this work are summarized as follows:

• We formulate reliability-aware knowledge distillation as a practical learning problem for compact robotic visual perception under uncertain visual conditions. Instead of assuming that teacher predictions are uniformly reliable, we explicitly analyze the risk of negative transfer caused by biased or incorrect teacher supervision.
• We propose a reliability-aware knowledge distillation framework that separates teacher predictions into reliable and unreliable components according to their consistency with ground-truth labels. Reliable predictions are preserved as stable supervision, while unreliable predictions are corrected through ground-truth-constrained adaptive fusion.
• We design a Self-Evolving Knowledge Bank (SEKB) to progressively refine unreliable supervision during training. Unlike static correction strategies, SEKB maintains sample-level evolving targets and updates them through temporal accumulation and student-guided anchoring, allowing the student model to gradually improve supervision quality.
• We introduce a consistency regularization term to stabilize the evolution trajectory of corrected supervision. Extensive experiments on multiple datasets and heterogeneous teacher–student architectures validate the effectiveness of the proposed framework, indicating its potential for robust and efficient visual perception in resource-constrained robotic systems.

Although the motivation of this work is rooted in robotic perception under uncertain and dynamic environments, the current experiments are designed to isolate and evaluate a specific source of unreliability, namely erroneous teacher supervision during knowledge transfer. Therefore, the robustness discussed in this paper primarily refers to the student model’s resistance to negative transfer from unreliable teacher predictions. The present study does not directly evaluate robustness to explicit sensor corruption, occlusion, domain shift, continuous visual disturbances, or closed-loop robotic operation. Evaluating the proposed framework under these practical conditions is left for future work.

2 Related work

Knowledge distillation (KD) (Hinton et al., 2015) is a classical paradigm for model compression and knowledge transfer, originally introduced by Hinton et al. The core idea is to train a compact student model by mimicking the softened probability outputs of a larger teacher model through temperature scaling, enabling the student to capture not only hard label supervision but also the dark knowledge encoded in the teacher’s prediction distribution. This seminal work laid the foundation for subsequent advances in KD and has been widely adopted in various visual tasks.

Building upon this framework, later studies have systematically extended distillation mechanisms. Recent survey papers (Moslemi et al., 2024) categorize KD approaches into response-based, feature-based, and relation-based distillation, and further summarize emerging paradigms such as self-distillation and cross-modal distillation. These methods have achieved notable improvements across diverse visual recognition tasks. However, most existing KD approaches implicitly assume that teacher predictions are globally reliable, overlooking the presence of unreliable or erroneous knowledge in the teacher outputs, which may lead to negative transfer.

Recent studies have increasingly recognized that teacher predictions are not equally reliable across all samples and categories. Influenced by factors such as label noise, class imbalance, and limited model capacity, teacher models often produce systematic errors on specific samples, particularly in long-tailed scenarios. When such predictions are indiscriminately transferred as supervision signals, the student may inherit both correct and incorrect knowledge, leading to performance degradation.

To address this issue, several studies propose class-balanced or distribution-aware distillation strategies, such as reweighting distillation losses or decoupling representation learning from classification, to mitigate the adverse impact of teacher bias (Tong et al., 2025; Kang et al., 2019; Iscen et al., 2021). In addition, some research investigates distillation reliability from the perspectives of robust learning and noisy supervision. These methods treat teacher outputs as potentially noisy labels and employ instance reweighting or robust loss formulations to suppress the influence of erroneous predictions (Huang et al., 2023; Jiang et al., 2023; Arazo et al., 2019; Han et al., 2018). Under more challenging conditions, such as adversarial perturbations or distribution shifts, curriculum-based or stage-wise distillation strategies have been explored to improve training stability (Li et al., 2023; Liu et al., 2023).

Despite these efforts, most existing approaches primarily focus on reducing the impact of unreliable predictions through weighting or suppression, while lacking explicit mechanisms to correct erroneous knowledge or enable adaptive refinement during training. As a result, students are often treated as passive recipients rather than active learners capable of summarizing and refining transferred knowledge.

Self-distillation has recently emerged as an important direction in knowledge distillation, aiming to improve model performance without relying on additional external teacher models. Early studies such as Born-Again Neural Networks (Furlanello et al., 2018) demonstrate that even identical architectures can benefit from distillation, suggesting that models are capable of reorganizing and reusing their own knowledge.

Subsequent work has explored multi-stage and multi-generation distillation frameworks, where predictions from earlier training stages or previous model generations are used to supervise later ones (Zhang et al., 2025; Tong et al., 2026). In parallel, online distillation and mutual learning paradigms enable collaborative learning among multiple student models, facilitating dynamic knowledge interaction and refinement (Zhang et al., 2018; Yang et al., 2023). From a broader perspective, recent studies have revisited distillation from the viewpoint of knowledge evolution, where techniques such as exponential moving average (EMA) teachers provide more stable and consistent supervision signals (Tarvainen and Valpola, 2017; Lavin, 2023).

Nevertheless, existing self-distillation and evolutionary distillation methods primarily exploit temporal consistency or architectural redundancy, while rarely explicitly distinguishing reliable and unreliable knowledge during the distillation process. Consequently, erroneous knowledge is not explicitly corrected, and the student’s ability to refine and evolve knowledge remains underexplored. This limitation motivates the development of reliability-aware and evolution-driven distillation frameworks, as proposed in this work.

3 Methods

3.1 Problem formulation and overview

We consider a deployment-oriented robotic visual perception scenario, where a compact student model is trained to approximate and refine the knowledge of a high-capacity teacher model for efficient inference. In the knowledge distillation setting, we consider a labeled training dataset D = { ( x i , y i ) } i = 1 N, where x i denotes the input sample, y i ∈ { 1 , … , C } denotes the corresponding ground-truth class index, and y i ∈ { 0,1 } C denotes its one-hot representation of y i. A pre-trained teacher model T produces a class probability distribution = p i T for each input. The goal is to train a lightweight student model S that retains the discriminative ability of the teacher while improving robustness and deployment efficiency for compact visual perception.

Conventional knowledge distillation methods typically assume that the teacher’s predictions are reliable for all samples, and thus directly transfer the teacher’s soft outputs to the student as supervision. However, in complex real-world visual recognition scenarios, the teacher model may produce incorrect or biased predictions for certain samples. When all teacher knowledge is indiscriminately distilled, the student may inherit not only the teacher’s strengths but also its errors, leading to performance degradation or negative transfer.

To address this issue, we propose a reliability-aware knowledge distillation framework with self-evolving supervision. As illustrated in Figure 1, the proposed framework consists of three key stages: reliability identification, knowledge correction, and self-evolving supervision refinement. Teacher predictions are first categorized into reliable and unreliable components. Unreliable supervision is then rectified and stored in a Self-Evolving Knowledge Bank (SEKB), which is selectively updated across training stages from t to t + 1. Through anchor-guided fusion and temporal accumulation, the evolving knowledge progressively integrates teacher experience, student refinement, and ground-truth constraints, forming a stable supervision trajectory over time.

Intuitively, the proposed framework follows an “identify–correct–evolve” workflow. Reliability identification first separates teacher predictions according to their consistency with the ground-truth labels. Reliable predictions are retained as stable supervision, while unreliable predictions are converted into ground-truth-constrained soft targets through knowledge correction. SEKB then progressively refines these sample-specific targets by integrating historical knowledge, current student predictions, and ground-truth constraints. Unlike static correction, SEKB updates the targets throughout training; unlike conventional temporal averaging or EMA-based teacher updating, it selectively evolves the supervision of unreliable samples rather than averaging raw predictions or updating teacher parameters.

3.2 Reliable and unreliable knowledge identification

In the knowledge distillation process, teacher predictions do not exhibit uniform reliability across all training samples. To prevent erroneous teacher knowledge from being directly transferred to the student model, we first perform reliability identification on teacher predictions.

Specifically, for each training sample x i, the teacher model produces a class probability distribution p i T, and the predicted class is given by Equation 1:

y ̂ i T = arg max c p i , c T . (1)

By comparing the teacher’s predicted label with the ground-truth label y i, we categorize teacher knowledge into two types:

• Reliable Knowledge: if y ̂ i T = y i, the teacher prediction matches the ground truth, and the corresponding output is regarded as reliable knowledge;
• Unreliable Knowledge: if y ̂ i T ≠ y i, the teacher prediction is inconsistent with the ground truth, and the corresponding output is considered unreliable knowledge.

This identification strategy is simple, interpretable, and does not rely on additional confidence estimation or heuristic thresholds. By explicitly distinguishing the reliability of teacher knowledge at an early stage, our framework establishes a clear foundation for subsequent knowledge correction and evolution.

It is important to note that unreliable knowledge is not discarded in our framework. Instead, it is treated as a supervision signal that may contain useful discriminative information but suffers from bias. In the following stages, we introduce dedicated correction and knowledge evolution mechanisms to refine and exploit such knowledge effectively.

For samples identified as reliable, the teacher prediction is directly used as a soft supervision target. Let R denote the set of reliable samples, where y ̂ i T = y i. The reliability-aware distillation loss is defined in Equation 2:

L rel = 1 | R | ∑ i ∈ R K L p i T ‖ p i S , (2)

where p i T and p i S denote the teacher and student probability distributions for sample x i, respectively. This term encourages the student to inherit teacher knowledge only when the teacher prediction is consistent with the ground-truth label, thereby reducing the risk of transferring erroneous supervision.

3.3 Knowledge correction for unreliable predictions

After identifying the reliability of teacher predictions, directly distilling unreliable teacher outputs would inevitably propagate erroneous knowledge to the student model. To alleviate this issue, we introduce a knowledge correction mechanism specifically designed for unreliable predictions, which preserves potential discriminative information from the teacher while aligning the supervision signal with the ground-truth semantic space.

For each unreliable sample x i, where y ̂ i T ≠ y i, we construct a corrected teacher knowledge representation p i ̃ in Equation 3:where p i T denotes the class probability distribution predicted by the teacher model, y i denotes the one-hot representation of the ground-truth class index y i, and λ i ∈ [ 0,1 ] is a sample-adaptive weighting factor.

p ̃ i = λ i p i T + 1 − λ i y i , (3)

To quantify how much teacher information should be retained for an unreliable sample, we compute a sample-adaptive coefficient according to the teacher confidence on the ground-truth class. Let z i T denote the teacher logits before softmax. The coefficient is defined as

λ i = exp z i , y i T / τ ∑ c = 1 C exp z i , c T / τ , (4)

where z i , y i T is the teacher logit corresponding to the ground-truth class y i, and τ is a temperature parameter. In this way, λ i reflects the relative confidence assigned by the teacher to the correct class. A larger λ i indicates that the teacher still contains useful information about the ground-truth class even though its top-1 prediction is incorrect, while a smaller λ i makes the corrected target rely more on the ground-truth label. The coefficient λ i is calculated independently for each unreliable sample and therefore adaptively balances the teacher distribution and the ground-truth supervision.

This formulation provides an intuitive interpretation: when the teacher assigns relatively high confidence to the correct class despite making an incorrect prediction, its output still contains useful information; otherwise, the corrected knowledge relies more heavily on ground-truth supervision. Through this correction process, unreliable teacher knowledge is continuously projected into a ground-truth-constrained soft supervision space, laying a stable foundation for subsequent knowledge evolution and distillation.

3.4 Self-evolving knowledge bank

Although the knowledge correction mechanism described above can alleviate the negative influence of unreliable teacher predictions, it remains a static refinement process and is insufficient to capture the student’s progressively enhanced discriminative capability during long-term training. To this end, we further introduce a Self-Evolving Knowledge Bank (SEKB), which continuously aggregates and updates supervision signals throughout training, enabling the student model to achieve stable improvements in regions where the teacher exhibits limited competence.

Specifically, for each training sample x i, we maintain an evolving knowledge representation q i ( t ) ∈ R C, where t denotes the number of updates performed on the knowledge-bank entry associated with sample i. Hereafter, evolving knowledge refers specifically to q i ( t ), while knowledge evolution denotes its recursive update process in SEKB; together, these updates provide the self-evolving supervision used to train the student. The representation q i ( t ) encodes the soft supervisory knowledge accumulated from previous visits to sample i. At the initial stage, the knowledge bank is initialized using the corrected teacher knowledge obtained in Section 3.3, given by Equation 5:where p i T denotes the teacher’s prediction distribution and p ̃ i represents the corrected knowledge constrained by ground-truth supervision. This initialization ensures that reliable samples directly inherit the teacher’s strengths, while unreliable samples are anchored to a semantically consistent starting point before entering the evolution stage.

q i 0 = p i T , y ̂ i T = y i p ̃ i , y ̂ i T ≠ y i , (5)

During subsequent training, the evolving knowledge q i ( t ) is updated according to the learning dynamics of the student model. When an unreliable sample x i is encountered in a mini-batch for its tth knowledge-bank update, the current student prediction p i S , ( t ) is used to construct an anchor target that jointly incorporates student predictions and ground-truth supervision:where y i denotes the one-hot representation of the ground-truth class index y i, and μ ∈ [ 0,1 ] balances the contribution of the student prediction and ground-truth supervision. The evolving knowledge is then updated via a temporal recursion:where α ∈ [ 0,1 ] controls the retention of historical knowledge. This update rule enforces temporal consistency while allowing the evolving knowledge to progressively integrate student-driven information under ground-truth anchoring, thereby preventing drift toward incorrect class regions.

a i t = μ p i S , t + 1 − μ y i , (6)

q i t + 1 = α q i t + 1 − α a i t , (7)

Unlike conventional EMA-based teacher updating or temporal ensembling, SEKB selectively updates only unreliable samples and evolves ground-truth-anchored soft targets rather than model parameters or raw prediction averages. Therefore, it functions as a reliability-guided correction memory. For reliable samples, the original teacher predictions are preserved, while SEKB focuses on unreliable samples where teacher supervision requires correction.

In implementation, SEKB is maintained as a sample-indexed buffer. Each training sample is assigned a fixed dataset index that remains unchanged under data shuffling. During each mini-batch, only the SEKB entries corresponding to unreliable samples are retrieved and updated. Under the standard shuffled epoch-wise training protocol, each sample is typically visited once per epoch, so each unreliable sample-specific entry is effectively updated once per epoch.

The historical bank entry q i ( t ) is stored as a non-trainable buffer and is detached from the computational graph. During the current forward pass, the student prediction p i S , ( t ) remains differentiable when constructing the anchor target a i ( t ) and the temporary updated target q i ( t + 1 ). Consequently, gradients from the evolution and consistency losses are propagated to the current student prediction, while no gradient is propagated through the historical bank state or across previous mini-batches. After the losses are computed, a detached copy of q i ( t + 1 ) is written back to the corresponding SEKB entry for use when sample i is visited again.

SEKB stores one C-dimensional probability vector for each training sample, resulting in a memory complexity of O ( N C ), where N is the number of training samples and C is the number of classes. Assuming 32-bit floating-point storage, the corresponding memory requirements are approximately 1.9 MiB for CIFAR-10, 19.1 MiB for CIFAR-100, and 76.3 MiB for Tiny-ImageNet. For a mini-batch containing B u unreliable samples, the update requires O ( B u C ) element-wise operations. SEKB introduces no additional trainable parameters and requires no additional forward pass through either the teacher or student network.

From an optimization perspective, the evolving knowledge q i ( t ) serves as an additional soft supervision signal for student training. For unreliable samples, the student is guided by minimizing the Kullback–Leibler divergence between its prediction distribution and the evolving knowledge (Equation 8):

L evo = 1 | U b | ∑ i ∈ U b KL q i t ‖ p i S , t . (8)

To further stabilize the evolution process, we introduce a consistency regularization term that penalizes abrupt changes in the evolving knowledge across consecutive stages (Equation 9):Here, U b denotes the set of unreliable samples in the current mini-batch. Both L evo and L cons are averaged over the samples in U b and are set to zero when U b is empty. Through iterative updates, SEKB gradually integrates teacher knowledge, student predictions, and ground-truth constraints, producing more stable supervision than single-step correction.

L cons = 1 | U b | ∑ i ∈ U b q i t + 1 − q i t 2 2 , (9)

3.5 Training objective and optimization

Building upon the components introduced in Sections 3.2–3.4, we present the overall training objective and optimization procedure for the student model.

The learning process is jointly guided by ground-truth supervision, reliable teacher knowledge distillation, and self-evolving supervision provided by the self-evolving knowledge bank. The overall training objective is formulated aswhere L C E is the standard cross-entropy loss with ground-truth labels, L rel transfers reliable teacher knowledge on samples in R, L evo guides the student using the evolving knowledge stored in SEKB for samples in U, and L cons stabilizes the temporal evolution of the knowledge bank. The coefficients β, γ, and η control the relative importance of these terms.

L = L C E + β L rel + γ L evo + η L cons , (10)

During optimization, the teacher model is kept fixed. The reliable and unreliable sets are determined based on the consistency between teacher predictions and ground-truth labels. For reliable samples, the student learns from both hard labels and teacher soft predictions. For unreliable samples, direct distillation from the teacher is avoided; instead, the student is supervised by corrected and progressively evolved targets. This design prevents erroneous teacher predictions from being directly propagated while still preserving useful distributional information.

To provide a clearer description of the overall optimization process, the training procedure of the proposed framework is summarized in Algorithm 1. The teacher model is first used to identify reliable and unreliable supervision according to the consistency between teacher predictions and ground-truth labels. Reliable knowledge is directly distilled to the student, while unreliable knowledge is corrected and progressively refined through the Self-Evolving Knowledge Bank (SEKB). In each training epoch, the student model is optimized under the joint supervision of ground-truth labels, reliable teacher knowledge, and evolving corrected knowledge.

Algorithm 1 Overall training procedure of the proposed framework.

4 Experiment

4.1 Experimental setup

We evaluate the proposed method on three widely used image classification benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet. CIFAR-10 (Krizhevsky, 2009) and CIFAR-100 (Krizhevsky, 2009) consist of 60,000 color images of size 32 × 32 from 10 to 100 classes, respectively, with 50,000 images for training and 10,000 images for testing. In addition, we conduct experiments on the Tiny-ImageNet (Russakovsky et al., 2015) dataset, which contains 200 object categories with 100,000 training images, 10,000 validation images, and 10,000 test images. Compared with CIFAR datasets, Tiny-ImageNet provides a more challenging yet computationally affordable benchmark, enabling a broader evaluation of knowledge transfer effectiveness across a larger number of visual categories.

To comprehensively assess the architectural generality of the proposed approach, we conduct experiments using a diverse set of teacher–student network architectures. Specifically, we consider three representative model families: convolutional neural networks (ResNet-50 (He et al., 2016), ResNet-18 (He et al., 2016), and MobileNet-V2 (Sandler et al., 2018)), vision transformers (Dosovitskiy et al., 2021) (ViT-Base and ViT-Tiny), and recent sequence-based visual models (Zhu et al., 2024) (ViM-Base and ViM-Tiny). These models span a wide range of architectural principles and model capacities, enabling evaluation under both homogeneous and heterogeneous distillation settings.

Regarding the experimental protocol, the official training and test splits were used for CIFAR-10 and CIFAR-100. For Tiny-ImageNet, the official training split was used for model training, while the labeled validation split was used for final evaluation; the unlabeled official test split was not used. Tiny-ImageNet images were resized from 64 × 64 to 224 × 224 to provide a unified input resolution for all CNN, Transformer, and sequence based architectures evaluated in this study. The teacher models were first trained on the corresponding datasets and then kept fixed during distillation, while the student models were initialized from scratch. For all datasets and model architectures, we adopted the standard preprocessing and data-augmentation settings of the corresponding official implementations without introducing additional task-specific augmentation. Under each teacher–student configuration, the same model initialization, preprocessing, augmentation, and training schedule were applied to the student baseline, all competing methods, and the proposed method.

We compare the proposed method with eight representative knowledge distillation baselines, including KD (Hinton et al., 2015), RKD (Park et al., 2019), CRD (Tian et al., 2020), DKD (Zhao et al., 2022), LSKD (Sun et al., 2024), AFD (He et al., 2025), BKD (Zhang et al., 2023), and LDN (Shen et al., 2025). These methods cover diverse distillation paradigms such as logit-based, relation-based, contrastive, decoupled, and confidence-aware distillation. For fair comparison, all baseline methods are evaluated under the same backbone architectures, datasets, data preprocessing, training epochs, batch size, and optimizer settings as the proposed method. Method-specific hyper-parameters are set according to the original papers or official implementations whenever available. All models are optimized using the Adam optimizer with an initial learning rate of 1 × 1 0 − 4. Unless otherwise specified, all models are trained for 300 epochs with a fixed batch size of 128.

The hyperparameters of the proposed method were kept fixed across all datasets and teacher–student architectures. Specifically, the temperature parameter in Equation 4 was set to τ = 0.3, the student–label fusion coefficient in Equation 6 was set to μ = 0.7, and the historical knowledge retention coefficient in Equation 7 was set to α = 0.6. The loss weights in Equation 10 were set to β = γ = η = 1. These global coefficients were empirically fixed to provide a stable balance among the corresponding components and were not tuned separately for individual datasets or architectures. The coefficient λ i was calculated independently for each unreliable sample according to Equation 4 and was therefore not manually specified. The temperature τ was used only to calculate λ i, without additional temperature scaling in the KL-divergence terms. During mini-batch optimization, L evo and L cons were averaged over the unreliable samples in the current mini-batch and were set to zero when no unreliable sample was present. The reliable distillation loss was computed as KL ( p i T ‖ p i S ), while the evolution loss was computed as KL ( q i ( t ) ‖ p i S , ( t ) ), with the teacher-derived or evolving distribution serving as the supervision target. Each experiment is conducted three times, and the average Top-1 accuracy is reported.

4.2 Main results

Results on Tiny-ImageNet (CNN-based Distillation). Table 1 reports the top-1 classification accuracy of different knowledge distillation methods on Tiny-ImageNet using ResNet-50 as the teacher model. The student networks include ResNet-18 and MobileNet-V2.

TABLE 1. Top-1 accuracy (%) of CNN-based knowledge distillation methods on Tiny-ImageNet. The teacher model is ResNet-50.

As shown in Table 1, the vanilla student models trained without distillation achieve 54.60% and 55.06% accuracy for ResNet-18 and MobileNet-V2, respectively. Conventional logit-based distillation (KD) provides modest improvements over the student baseline. More advanced methods, such as DKD and LSKD, further enhance performance by explicitly modeling logit decoupling and structured supervision.

Compared with these representative baselines, the proposed method consistently achieves the best performance across both student architectures. Specifically, our approach improves ResNet-18%–68.31%, and MobileNet-V2 to 67.33%. Notably, the proposed method not only surpasses all compared distillation baselines, but also outperforms the teacher model (65.41%) by a clear margin.

We attribute this improvement to the reliability-aware knowledge selection mechanism, which explicitly distinguishes reliable supervisory signals from unreliable ones during training. By adaptively preserving stable knowledge while correcting inconsistent predictions, the student model is able to achieve better generalization performance than directly inheriting the teacher outputs. These results demonstrate that the proposed framework effectively enhances knowledge transfer under medium-scale visual perception settings.

Results on Tiny-ImageNet (Transformer-based Distillation). Table 2 presents the results when using ViT-Base as the teacher model. We evaluate both homogeneous distillation (ViT-Base → ViT-Tiny) and heterogeneous distillation (ViT-Base → ResNet-18).

TABLE 2. Top-1 accuracy (%) of Transformer-based knowledge distillation methods on Tiny-ImageNet. Results include both homogeneous distillation (ViT-Base → ViT-Tiny) and heterogeneous distillation (ViT-Base → ResNet-18).

For the homogeneous setting, conventional distillation methods provide moderate improvements over the ViT-Tiny baseline. Methods such as DKD, LSKD, and BKD achieve gradual gains, indicating that structural supervision and refined logit modeling are beneficial for transformer-based students. The proposed method further improves ViT-Tiny to 73.06%, achieving consistent gains over all compared baselines.

In the heterogeneous setting (ViT-Base → ResNet-18), the performance gap between teacher and student architectures introduces additional transfer difficulty. Nevertheless, our method improves ResNet-18%–72.81%, demonstrating strong cross-architecture knowledge transfer capability. The consistent improvements across both homogeneous and heterogeneous scenarios verify that the proposed reliability-aware distillation strategy is not restricted to convolutional inductive biases, but generalizes effectively to transformer-based representations.

Results on Tiny-ImageNet (Sequence-based Distillation). Table 3 reports the results using ViM-Base as the teacher model. Similar to the transformer setting, we evaluate both homogeneous (ViM-Base → ViM-Tiny) and heterogeneous (ViM-Base → ResNet-18) distillation.

TABLE 3. Top-1 accuracy (%) of sequence-based knowledge distillation methods on Tiny-ImageNet. Results include both homogeneous distillation (ViM-Base → ViM-Tiny) and heterogeneous distillation (ViM-Base → ResNet-18).

Compared with the student baseline, existing distillation methods provide steady improvements in both settings. However, the proposed method consistently achieves superior performance. In particular, ViM-Tiny is improved to 72.00%, and ResNet-18 reaches 70.19%, demonstrating that the proposed approach remains effective even when the teacher model is based on state-space representations.

These results indicate that the proposed reliability-guided knowledge transfer mechanism is architecture-agnostic and can adapt to different representation paradigms, including convolutional networks, transformers, and state-space models.

Results on CIFAR-10 and CIFAR-100. Table 4 presents the results on CIFAR-10 and CIFAR-100 using ResNet-50 as the teacher model. Compared with Tiny-ImageNet, CIFAR datasets are smaller in scale but widely adopted for evaluating knowledge distillation methods under controlled experimental settings.

TABLE 4. Top-1 accuracy (%) of CNN-based knowledge distillation methods on CIFAR-10 and CIFAR-100. The teacher model is ResNet-50 (96.08% on CIFAR-10% and 79.34% on CIFAR-100). Results include both homogeneous and lightweight student architectures.

On CIFAR-10, the vanilla student models achieve 93.30% (ResNet-18) and 90.08% (MobileNet-V2), respectively. Existing distillation methods consistently improve the student performance, with LDN reaching 95.89% and 95.80%. The proposed method further enhances the performance to 96.67% and 96.63%, surpassing both the baseline methods and the teacher model (96.08%).

On CIFAR-100, which is more challenging due to the larger number of categories, the improvements brought by advanced distillation methods become more evident. While the strongest baseline achieves around 75.6%, the proposed method significantly improves the student performance to 81.50% (ResNet-18) and 81.47% (MobileNet-V2). These results demonstrate that the proposed reliability-aware distillation framework remains effective across datasets of different scales and complexity levels.

Overall, the consistent improvements observed on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate the effectiveness of the proposed method under unreliable teacher supervision. The gains are maintained across different student capacities and architectural paradigms, highlighting the stability and architectural generality of the proposed knowledge transfer framework.

4.3 Ablation studies

To rigorously assess the contribution of each component in the proposed framework, we conduct a series of ablation experiments on Tiny-ImageNet using ResNet-50 as the teacher and ResNet-18 as the student. The results are reported in Tables 5–7. All experiments are conducted under identical training settings to ensure fair comparison.

TABLE 5. Overall ablation of the proposed framework on Tiny-ImageNet (ResNet-50 → ResNet-18). The effects of reliability identification, knowledge correction, and the Self-Evolving Knowledge Bank (SEKB) are evaluated.

TABLE 6. Component analysis of the Self-Evolving Knowledge Bank (SEKB) on Tiny-ImageNet (ResNet-50 → ResNet-18).

TABLE 7. Ablation of different loss components on Tiny-ImageNet (ResNet-50 → ResNet-18).

4.3.1 Overall framework analysis

Table 5 presents the module-level ablation study of the proposed framework. The baseline student trained only with cross-entropy achieves 54.60% Top-1 accuracy, while conventional KD improves the performance to 56.10%. This result indicates that directly transferring soft teacher predictions provides only limited gains when unreliable teacher supervision is not explicitly handled.

When reliability identification is introduced, the accuracy increases to 60.37%. In this setting, teacher predictions are first divided into reliable and unreliable samples according to their consistency with ground-truth labels, and only reliable teacher predictions are used as soft supervision. The improvement demonstrates that explicitly filtering out unreliable teacher outputs can effectively reduce negative transfer and provide more stable supervision for the student model.

After further introducing knowledge correction, the accuracy reaches 58.99%. In this variant, unreliable teacher predictions are corrected by ground-truth-constrained fusion and then used for student training. Although this strategy allows the student to exploit information from previously discarded unreliable samples, the performance is slightly lower than using reliability identification alone. This suggests that one-step static correction may introduce over-adjusted or insufficiently stable supervision for some ambiguous samples when no progressive refinement mechanism is used.

When SEKB is combined with reliability identification, the performance increases significantly to 65.27%. This result verifies the importance of self-evolving supervision. Different from one-step distillation, SEKB enables multi-stage knowledge refinement by progressively updating supervision signals during training. In this ablation setting, unreliable samples are refined through SEKB without static correction, which demonstrates that dynamic evolution itself can substantially improve the utilization of uncertain supervision.

Finally, the full model achieves the best accuracy of 68.31%. Compared with the variant using SEKB alone, the full model further benefits from knowledge correction, which provides a more reliable initialization for unreliable samples before knowledge evolution. These results show that the three components play complementary roles: reliability identification prevents direct transfer of erroneous teacher predictions, knowledge correction provides ground-truth-constrained supervision for unreliable samples, and SEKB progressively refines the corrected supervision through multi-stage evolution. Their combination leads to the most effective and stable knowledge transfer.

4.3.2 Component analysis of SEKB

To further examine the structural validity of the Self-Evolving Knowledge Bank (SEKB), we decompose it into its three core components: selective updating, temporal accumulation, and anchor-guided fusion. The results are reported in Table 6.

When the selective updating mechanism is removed and all samples are allowed to evolve, performance drops sharply to 57.30%. This significant degradation indicates that evolving already reliable samples can disturb stable knowledge representations. The selective mechanism therefore serves as an important safeguard, preventing unnecessary perturbation of well-learned supervision signals.

Removing the temporal update mechanism, which eliminates recursive accumulation across training iterations, reduces performance to 59.34%. This result highlights the importance of progressive refinement. Without temporal accumulation, evolution degenerates into a single-step adjustment, losing its capability to gradually refine unreliable predictions. The effectiveness of SEKB thus relies on knowledge being accumulated and stabilized over time rather than applied instantaneously.

Similarly, excluding the anchor target fusion, thereby removing the student’s participation in constructing the evolution target, leads to a decrease in performance to 60.23%. In this configuration, evolution becomes guided solely by teacher-derived information, resembling a smoothed teacher signal. The performance gap suggests that incorporating the student’s current representation into the evolution process is crucial for adaptive refinement and mutual teacher–student interaction.

The consistent degradation observed across all three variants confirms that SEKB cannot be reduced to a simple smoothing or averaging mechanism. Its effectiveness arises from the coordinated interaction of selective updating, temporal accumulation, and anchor-guided refinement. Each component plays a structurally indispensable role in enabling stable and adaptive knowledge evolution.

4.3.3 Loss component analysis

We further investigate the contribution of each loss term in Table 7. Using only the cross-entropy loss yields 54.60%, corresponding to the student baseline. Adding the reliability-aware supervision term increases performance to 61.41%, demonstrating that selectively distilling reliable teacher predictions can effectively reduce the influence of erroneous supervision and improve training stability.

Introducing the evolution loss further raises performance to 64.47%. This improvement indicates that distribution-level refinement of unreliable samples provides additional benefits beyond reliability reweighting alone. The evolution loss encourages progressive correction of uncertain predictions, thereby enhancing representation quality.

Finally, incorporating the consistency constraint results in the full model performance of 68.31%. The substantial gain from 64.47% to 68.31% underscores the importance of stabilizing the evolution trajectory. By constraining temporal deviations between successive refinement steps, the consistency term ensures smooth knowledge accumulation and prevents unstable oscillations.

Overall, the three loss components exhibit complementary roles. The reliability term improves supervision quality, the evolution term enables adaptive refinement, and the consistency term guarantees stable convergence. Their joint optimization is therefore necessary to achieve the full potential of the proposed framework.

5 Conclusion

In this paper, we proposed a reliability-aware knowledge distillation framework with self-evolving supervision for robust robotic visual perception. Different from conventional distillation methods that directly transfer all teacher predictions, the proposed approach explicitly separates reliable and unreliable supervision. Reliable predictions are preserved as stable soft targets, while unreliable predictions are corrected under ground-truth constraints and progressively refined through the Self-Evolving Knowledge Bank (SEKB). This design reduces the risk of negative transfer and enables the student model to learn from more reliable and adaptive supervision.

Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the proposed framework consistently improves compact student models over representative distillation baselines across convolutional networks, vision transformers, and sequence-based visual models. These results demonstrate the effectiveness of the proposed method in mitigating negative transfer caused by unreliable teacher supervision. Accordingly, the robustness investigated in the present study primarily refers to the robustness of knowledge transfer against erroneous teacher predictions, rather than comprehensive robustness to environmental disturbances such as sensor noise, image corruption, occlusion, domain shift, or closed-loop robotic disturbances.

From a practical perspective, the proposed framework can be integrated into a robotic perception pipeline as an offline training module. A high-capacity teacher model can provide supervision using labeled robotic or task-specific visual data, while reliability identification, knowledge correction, and SEKB progressively improve the supervision used to train a compact student model. The resulting student can then be deployed as a visual backbone or task-specific perception model on resource-constrained robotic platforms. Nevertheless, the current study is limited to image-classification benchmarks and does not directly evaluate explicit environmental disturbances or closed-loop robotic tasks. Future work will extend the framework to object detection, semantic segmentation, continuous visual perception, corruption and domain-shift settings, and real robotic platforms.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Chaoqun Wang, Shandong University, China

Reviewed by: Ruyu Liu, Technical University of Denmark, Denmark

Jiaming Lu, Hohai University, China