TCNNISTD: a parallel transformer–CNN network for infrared small target detection

1 Introduction

Infrared detectors, which capture and image the infrared radiation of objects, are endowed with many salient advantages, such as an extensive detection range, all-weather operability, and resilience to lighting and atmospheric conditions. These attributes enable infrared detectors to function effectively in adverse weather scenarios, including rain, snow, fog, and nighttime (Liu et al., 2023b). Consequently, these devices have garnered widespread application across diverse fields, including security surveillance, fire rescue, remote sensing, medical monitoring, and intelligent driving.

Within the domain of infrared detection systems, the detection of small infrared targets has perennially been a focal point of research. Small targets are typically defined as objects imaged from considerable distances, resulting in minimal pixel representation—often fewer than 10 pixels. Compared with larger targets, small targets are characterized by the absence of distinct shape features, a paucity of pixels, and a propensity to be obscured by system noise, all of which collectively exacerbate the challenges associated with their detection (Luo et al., 2024; Wang et al., 2025a).

At present, infrared small-target detection methods can be broadly bifurcated into traditional methods and learning-based approaches. Traditional algorithms hinge on the design of detection operators and optimization functions predicated on the characteristics of small targets. For example, methods that enhance local contrast and estimate background have been extensively explored. Du and Hamdulla (2020) proposed a strategy that leverages superpixel graphs to compute the saliency of small targets via homogeneity-weighted local patches. To suppress background clutter, Liu et al. (2023a) employed multidirectional gradients to accentuate the local contrast of small targets. Zhang et al. (2024) devised a diffusion difference measure (DDM) kernel to mitigate edge clutter in the background. While these methods exhibit commendable performance in typical infrared scenarios with simple background clutter, real-world applications often confront complex natural and artificial scenes with highly variable background noise, leading to frequent false alarms and missed detections.

In recent years, deep learning-based object detection algorithms have precipitated a paradigm shift in the field of computer vision, offering novel solutions for infrared small-target detection through their robust feature extraction capabilities. Guo et al. (2024) introduced semantic location information into a dense nested network to augment the network's attention to small targets. Yang et al. (2024) designed a network with multiangle pyramid feature fusion for detecting infrared small targets. Liu et al. (2024) integrated scale and location information into a U-Net to enhance the network's decision-making ability.

However, numerous challenges persist in the application of deep learning-based infrared small-target detection models. First, the targets to be detected are characterized by a small number of pixels, complex shapes, and a wide range of size variations, compounded by poor image quality, which collectively escalates the difficulty of training deep learning models. Second, infrared images often suffer from significant background noise and contrast issues, which substantially interferes with small-target detection. Existing deep learning object detection algorithms rely primarily on texture and geometric features to construct high-level semantics. However, infrared small targets are too small to provide distinct contour features, rendering these baseline deep learning algorithms unsuitable for direct application in small-target detection. To overcome these challenges and increase the detection rate of small targets in complex backgrounds, we propose TCNNISTD, a parallel dual-branch Transformer–CNN detection model. Unlike existing CNN-Transformer fusion methods that adopt serial or cascaded architectures, our parallel design allows each branch to preserve its native feature characteristics–local details from CNN and global context from Transformer—without cross-contamination caused by sequential processing. The contributions of this paper are as follows:

• We introduce a novel parallel dual-branch Transformer and CNN network (TCNNISTD) that synergistically amalgamates global dependencies from the Transformer branch and local detail features from the CNN branch, thereby optimizing detection accuracy. This parallel design fundamentally differs from serial fusion approaches by preserving modality-specific feature representations.
• A large-kernel CNN branch is designed by adapting RepLKNet specifically for infrared small target detection, employing a single 11 × 11 depthwise separable convolution as the first layer followed by 3 × 3 convolutions, which substantially expands the effective receptive field while avoiding over-smearing of tiny targets that plagues multi-layer large-kernel designs.
• Channel and spatial attention mechanisms are integrated into the Adaptive Fusion Module (AFM) to dynamically filter out background noise, empowering the model to focus more intently on salient small-target regions. Furthermore, a multi-scale loss supervision strategy is introduced to leverage hierarchical feature maps for improved segmentation performance.

The remainder of this paper is organized as follows. Section 2 provides a comprehensive overview of existing research. Section 3 elaborates on the key components and design principles of our proposed model. Section 4 details our experimental methodology and findings. Finally, Section 5 concludes this paper.

2 Related work

As related technologies continue to advance, research methods for infrared small target detection have diversified significantly and can be categorized into five primary types: filtering-based methods, human visual system (HVS)-based methods, multisubspace learning-based methods, low-rank and sparse decomposition-based methods, and deep learning-based methods, which have emerged in recent years.

(1) Filtering-based methods. Filtering-based techniques are extensively employed in infrared small target detection, enhancing the contrast between the target and background by applying various filters to improve detection performance. These methods are predicated on the assumption that infrared image backgrounds typically exhibit spatial consistency, which is disrupted by the presence of targets. Consequently, filters are utilized to suppress background noise and highlight the target. Common filtering techniques include the top-hat filter, max-median filter, and adaptive spatiotemporal filtering. Fan et al. (2017) developed an adaptive aperture spatiotemporal filtering method that dynamically adjusts filter parameters based on the motion characteristics of the target. Other researchers have also made improvements to filtering techniques. Huang et al. (2020) proposed a structure-adaptive clutter suppression method that employs chain-growth filtering technology to effectively mitigate background clutter, thereby enhancing small target detection in dynamic environments. Peng et al. (2019) utilized a shearlet transform-based kurtosis maximization method to detect infrared small targets under non-uniform backgrounds, achieving improved detection performance, especially in complex background scenarios. Hao et al. (2024) adaptively estimated target size via multidirectional gradient filtering and sinusoidal fitting, then suppressed clutter with an optimal morphological filter to detect IR small targets in complex scenes. While these methods achieve high detection accuracy in uniform backgrounds, their performance may significantly degrade in complex backgrounds or high-noise environments, leading to false alarms or missed detections. Factors such as background texture, lighting variations, and noise interference pose challenges to maintaining stable performance in practical applications.

(2) Human visual system (HVS)-based methods. HVS-based methods simulate the characteristics of the human visual system, such as scale adaptability, contrast sensitivity, and attention shift mechanisms, to detect small targets by calculating the local contrast Lu et al., (2023). Chen et al. (2014) employed a local contrast method (LCM) to describe the differences between a point and its neighborhood. Wei et al. (2016) used a multiscale relative local contrast measure for target detection, whereas Lu et al. (2022) introduced a novel small target detection method based on multidirectional derivative-weighted local contrast measurement (MDWCM). Tang et al. (2023) calculated the global contrast measure by dilate operation to suppress the background. Xu et al. (2024) designed a local measure combining both intensity and gradient information. However, these methods have notable limitations, including sensitivity to noise, and performance degradation in multitarget or heavily occluded environments. Although HVS-based methods exhibit significant potential, further optimization is required in terms of noise robustness to better address the challenges posed by complex scenarios and real-time applications.

(3) Multisubspace learning-based methods. Multiple subspace learning methods assume that the background is derived from multiple subspaces. The core of this approach is to model complex backgrounds more effectively by learning multiple subspaces and to detect targets by exploiting the differences in subspace representation between the target and background. Li et al. (2015) proposed a low-rank representation (LRR) model that uses the image itself as the input dictionary for modeling. He et al. (2015) considered the presence of noise and introduced a low-rank and sparse representation (LRSR) model, incorporating an overcomplete dictionary to sparsely represent the target and adding constraints to the noise component. Wang et al. (2017a) proposed a multisubspace learning (SMSL) method, and Zhang et al. (2021) introduced a self-regularized weighted sparse (SRWS) model. In these models, dictionary construction is particularly critical. However, both GIM and the use of the image itself as a dictionary have significant drawbacks. The VAG-SR model (Li et al., 2024) combines spatiotemporal structure preservation with visual attention guidance in a sparse reconstruction framework to effectively suppress clutter and detect IR small targets in complex scenes. The RPCA-based framework Zhang et al., (2026) uses a supervised discriminative dictionary and regularization to preserve structural information and extract small targets via sparse coding under complex backgrounds. SMSL methods can better handle background complexity and demonstrate good adaptability to different types of backgrounds, with relatively stable detection performance. However, they suffer from high computational complexity, poor real-time performance, and suboptimal performance in multitarget scenarios, leading to false alarms.

(4) Low-rank and sparse decomposition-based methods. Low-rank and sparse decomposition-based methods have demonstrated excellent performance in infrared small target detection. By treating the background as a low-rank matrix and the target as a sparse matrix, these methods effectively separate the two using optimization techniques. Inspired by the IPI model, many researchers have proposed improved algorithms. For example, Dai et al. (2017) proposed a method to construct the IPI model using partial sum minimization of singular values. Wang et al. (2017b) introduced total variation (TV) regularization to effectively suppress edge information. Zhang et al. (2018) utilized the L21 norm to approximate background clutter and introduced non-convex rank approximation minimization (NRAM). Zhang et al. (2019) proposed a model based on non-convex optimization with Lp norm constraint (NOLC) to better estimate the sparse target. Zhu et al. (2020) employed a logarithmic operator-based smooth but non-convex substitute for rank in the target-aware non-local low-rank (TANLR) model. Since the solution of the objective function requires iteration, although IPI-based methods have good performance, they have not yet achieved real-time performance and are difficult to apply in practical scenarios.

To handle high-dimensional data more efficiently, tensors are used to expand matrices Chen et al., 2020; Zheng et al., (2020). Zhang and Peng (2019) adopted a new non-convex low-rank constraint, the partial sum of the tensor nuclear norm (PSTNN), to effectively preserve the target. Guan et al. (2020) proposed a model combining non-convex tensor rank surrogate with local contrast energy (NTRS). Yang et al. (2020) proposed a new infrared cirrus detection model using Ket enhancement to represent tensor rank. Pang et al. (2025) integrate spatiotemporal and saliency priors into tensor decomposition, with tangent-space acceleration and multirank constraints, for efficient small-IR-target detection. Chen et al. (2026) develop a non-convex low-rank tensor completion approach with reweighted sparse regularization enables robust IR small-target detection by suppressing discrete noise while preserving target information. Overall, these methods have significant advantages in detection accuracy, but their high computational complexity and poor real-time performance need to be addressed to meet practical application requirements.

(5) Deep learning-based methods. Deep learning-based methods leverage deep learning models, such as convolutional neural networks (CNNs), to automatically learn the features of targets and backgrounds for target detection. In the field of infrared small target detection, Shi and Wang (2020) proposed a CDAE model that integrates a denoising autoencoder network and a CNN. The core idea is to focus on learning the background and treat small targets as “noise” in infrared images, thereby transforming the small target detection task into a denoising task. Liu and Meng (2020) proposed S-FCN, which uses convolutional networks for semantic segmentation and introduces attention mechanisms to improve detection accuracy. Dai et al. (2021b) introduced asymmetric context modulation in the UNet network, enabling interaction between high-level and low-level semantic features through the ACM module. Li et al. (2023) proposed DNANet, which embeds the CBAM module in the basic architecture. Although the detection performance is excellent, the network structure is relatively complex. Tong et al. (2021) proposed an efficient and powerful EAA attention module that combines channel attention (BAA) and spatial attention (SA) mechanisms for cross-fusion and cross-layer feature fusion. Wang et al. (2025b) introduce MAFNet, which combines multi-scale alignment fusion for fine-grained target perception and multi-field search for noise suppression, enabling robust IR small-target detection with low false-alarm rates. Li et al. (2025) integrate frequency-domain decomposition into the spatial encoder and gradient-based edge enhancement to effectively extract structural features and improve IR small-target detection.

However, due to low signal-to-noise ratio and complex backgrounds, these algorithms still suffer from missed detections and false alarms in practical applications. In summary, each infrared small target detection method has its own strengths and weaknesses and is suitable for different application scenarios.

Recently, Transformer-based architectures have gained increasing attention in infrared small target detection. Yuan et al. (2024) proposed SCTransNet, a spatial-channel cross Transformer network that models long-range dependencies via self-attention while preserving local features through cross-attention between encoder and decoder. Hu et al. (2024) introduced DATransNet, a dynamic attention Transformer network that adaptively weights multi-scale features for infrared small target detection. Additionally, UIU-Net Wu et al., (2023) incorporates a lightweight Transformer-like decoder nested within a U-shaped architecture to progressively refine target features, while ALCNet Dai et al., (2021b) leverages asymmetric context modulation to enhance multi-level feature interaction—both demonstrating the growing role of attention-driven feature fusion in this domain. Different from these approaches, our TCNNISTD adopts a parallel dual-branch design where the Transformer and CNN operate independently before fusion, rather than a serial arrangement where one follows the other. This parallelism preserves the distinct characteristics of both branches and mitigates the risk of small-target feature dilution in deep serial stacks. Furthermore, the dedicated large-kernel CNN branch in our framework provides a substantially larger effective receptive field than the standard 3 × 3 convolutions used in prior hybrid models.

Future research will focus on enhancing the robustness, real-time performance, and adaptability of detection algorithms to meet practical application requirements.

3 The proposed model

3.1 Overall framework

In recent years, researchers have developed numerous CNN-based network models and their improved algorithms for infrared small target detection. However, traditional CNNs face significant limitations in this task due to downsampling operations and deep encoders, which lead to a loss of local detail information. This issue is particularly pronounced in infrared small target detection, where targets often consist of only about ten or fewer pixels, and multiple convolutions greatly impact detection accuracy. Addressing this problem is crucial in designing effective infrared small target detection networks.

To tackle these limitations, we propose TCNNISTD, a parallel dual-branch architecture for infrared small target detection, as shown in Figure 1. This architecture includes a Transformer, a Large Kernel CNN, an Adaptive Fusion Module (AFM), a Detection Head, and a Segmentation Head. This architecture leverages the strengths of both Transformer and CNN: the Transformer's ability to learn global features benefits whole-image target detection, while the CNN's ability to learn local features aids in detecting small targets.

In the CNN branch, we use a large kernel convolution block followed by an improved res-block to extract multi-level features. In the Transformer module, we reshape the DeiT Transformer's output to convert sequence features into image features, forming the initial output feature. We then apply two upsampling blocks to generate multi-level features, maximizing the global characteristics in each scale transformer featuremap. The multi-scale outputs from the CNN and Transformer branches are fused through the AFM. Finally, the Detection Head and Segmentation Head detect and segment the infrared small target.

3.2 Transformer module

In order to improve the data efficiency of the training samples, we adopt DeiT (Touvron et al., 2021) as transformer module, which consists of Multi-head Self Attention layers (MSA) and Feed-Forward Network (FFN), as shown in Figure 2. MSA is an attention mechanism, based on a trainable associative memory with (key, value) vector pairs. Input a sequence of N query vector Q∈R^*N*×*d*, a set of k key vectors K∈R^*k*×*d*, a set of k value vectors V∈R^*k*×*d*, produce an output matrix (of size N×d) in Equation 1:

MSA(Q,K,V)=SoftMax(QKT/d)V.(1)

We add a FFN on top of the MSA layer. This FFN is composed of two linear layers separated by a GeLu activation. The first linear layer expands the dimension from d to 4d, and the second layer reduces the dimension from 4d back to d. Both MSA and FFN are operating as residual operators thanks to skip-connections, with layer normalization applied to each sub-layer. The input image size is 224 × 224, decomposed into a batch of N patch of a fixed size of 16 × 16, where N is equal to 14 × 14. Thus, after transformer module, we obtain a featuremap of 14 × 14. Then, we applied feature upsampling block, which composed of a Upsample layer and a convolution layer with kernel size of 3 × 3. The size of the feature map after upsampling block is 28 × 28 and 56 × 56, respectively.

3.3 Large kernel convolution module

Infrared small targets have few pixels and limited information, which is not enough to accomplish accurate target detection, and it is necessary to add scene context information to assist target detection. However, most of the existing CNN-based infrared small target detection networks use a resnet block with a convolutional kernel size of 3 × 3, which has a limited effective receptive field and cannot extract rich context information. For this reason, this paper investigates CNN networks using large convolutional kernels.

Inspired by the RepLKNet (Ding et al., 2022), we construct CNN branch via RepLK block. However, we modify the RepLK block so that it not only has a large receptive field, but also gives more attention on the small targets. Our RepLK module has the inception architecture, where 3 × 3, residual layer and large kernel k×k performed in a parallel way at the input and the output of these are added together to generate final output. After training, we adopt Structural Re-parameterization (Ding et al., 2021b, 2019, 2021a) methodology to merge the small kernel, residual layer as well as BN parameters into the large kernel, so the deploy model is equivalent to the model for training. In order to be able to compute quickly, we use depthwise separable convolution to achieve large kernel convolution. Specifically, standard convolution with a k×k kernel requires O(k2CinCoutHW) FLOPs, while depthwise separable convolution decomposes this into depthwise convolution O(k2CinHW) and pointwise convolution O(C_inC_outHW). For our 11 × 11 kernel with C_in = C_out, this reduces convolutional FLOPs by a factor of approximately k² = 121. The detailed Convolution branch is shown in Figure 3.

With the input image of size 224 × 224, we adopt RepLK block with k = 11 to generate output feature map of size 213 × 213. Then, a convolution kernel size of 3 × 3, stride = 2 to get the output of Stage1, which size is 112 × 112. Two RepLK blocks with k = 3 cascaded to extract the output of Stage2, Stage3 and Stage4, their dimensions are 56 × 56, 28 × 28 and 14 × 14 respectively, same size as the output feature map of the transformer branch.

3.4 Adaptive fusion module (AFM)

In the field of cognitive science, it is observed that the human visual system predominantly directs its focus toward the critical regions of an object. In recent years, researchers have applied the attention mechanism in the field of computer vision, which focuses on the important information in the image while others ignore irrelevant information. By assigning different weights to different parts of the input image to highlight the key information, we can see a performance improvement in target detection and semantic segmentation task.

For the infrared small target detection task, we should focus our attention on the small target area and its features, ignoring other information such as background and noise. Therefore, this paper incorporates an attention mechanism to enhance the precision and reliability of small target detection.

To dynamically integrate the multi-scale features from the CNN and Transformer branches, we propose the Adaptive Fusion Module (AFM), which combines channel attention and spatial attention in a cascaded manner.

TCNNISTD is composed of the Transformer branch and CNN branch in parallel. In the CNN branch, as the depth of downsampling increases, the size of the feature map becomes smaller, and the number of channels increases. This means that the model pays more attention to the semantic information on the channel. In the Transformer branch, the output feature map of the first layer is obtained after reshaping the sequence features into image features. Subsequently, two up-sampling operations are performed to get the output feature maps of the second and third layers. Although the number of channels decreases with upsampling, all three output feature maps can be directly output as a result. Therefore, in TCNNISTD, this paper uses the channel attention mechanism to deal with Transformer branches and CNN branches. The channel attention mechanism is a method for extracting the most representative channels in a feature map, which helps the network to better focus on important information while reducing the effect of noise. The structure of channel attention is shown in Figure 4.

Let Ficnn∈ℝC×Hi×Wi and Fitrans∈ℝC×Hi×Wi denote the feature maps from the CNN branch and Transformer branch at scale i∈{1, 2, 3, 4}, respectively. First, channel attention is applied independently to each branch in Equation 2:

Fica=Fi⊙σ(MLP(AvgPool(Fi))+MLP(MaxPool(Fi)))(2)

where ⊙ denotes element-wise multiplication and σ is the sigmoid function. The MLP consists of two fully-connected layers with reduction ratio r = 16.

In infrared small target detection, the introduction of channel attention can be utilized to weigh the feature weights of small targets with semantic information from the high-level feature semantic map. This design approach makes TCNNISTD better at focusing on the features of small targets. As a result, the background noise is eliminated without incurring additional computing costs, which lessens noise interference on the model and enhances its performance in the detection of infrared small target segmentation.

To make a connection between the feature maps of the Transformer branch and the CNN branch, the proposed model also employs the spatial attention mechanism. Unlike channel attention, spatial attention learns the importance of different spatial locations by attaching attention weights to the spatial dimensions of the feature map. Specifically, the spatial attention map is generated by computing the spatial relationship between the Transformer branch features and the CNN branch features. This approach effectively realizes multi-scale feature fusion, which in turn improves the performance of the model.

After channel attention, the branch features are concatenated in Equation 3:

Fifused=Fcnn,ica⊕Ftrans,ica(3)

where ⊕ denotes channel-wise concatenation. Finally, a spatial attention map is computed in Equation 4:

Fiout=Fifused⊙σ(f7·7([AvgPool(Fifused);MaxPool(Fifused)]))(4)

where f^7·7 is a 7·7 convolution with one output channel. The learnable parameters of AFM include the MLP weights W_ca and bias b_ca for channel attention, and the spatial convolution kernel Wsa∈ℝ1×1×7×7. This design learns content-adaptive fusion weights rather than using fixed summation or concatenation, enabling the network to dynamically emphasize salient small-target features while suppressing background noise.

The spatial attention mechanism can help the model to better focus on the spatial location of the target object in the image, instead of just focusing on the semantic information of the target object. This mechanism avoids the problems associated with ignoring spatial information when processing images and also enhances the feature weights of the target object. In the TCNNISTD model, the introduction of spatial attention enables more effective synergy between the Transformer branch and the CNN branch to improve the model's segmentation detection ability of the target to be detected.

3.5 Training details

As shown in Figure 1, after using attention fusion block to fuse multi-level feature maps, we designate Map3 as the primary prediction feature map for the infrared target and segmentation result, as ablation studies in Section 4.5 confirm that Map3 yields the best single-map performance. However, in order to fully utilize the information of the multi-scale features, as well as to make the fused features more characteristic on discriminating small targets, we propose a new training strategy, which introduces each fused feature map into the loss function and calculates loss separately. The loss function is defined in Equation 5:

F(X)=αL(map1)+βL(map2)+ωL(map3)+ηL(map4)(5)

where α, β, ω, and η are the loss function weight coefficients for map1, map2, map3, and map4, respectively. When one of the parameters is set 1 and the other parameters are 0, it denotes that the map corresponding to that parameter is used as the output, and the other maps are not involved in the loss function calculation.

The loss function L(x) contains Cross-entropy loss for detection and Dice Loss for segmentation, is defined in Equation 6.

L(x)=Lce(x)+γLdice(x)(6)

γ is a weight to balance the two loss terms. Dice Loss is a standard loss function for imbalanced classification, which is well-suited for datasets with highly skewed foreground-background distributions, defined in Equation 7:

DiceLoss(X)=1-2|X∩Y||X|+|Y|.(7)

Let Y represent the actual segmentation of the small target, and X represent the model's prediction result. Dice Loss takes a value ranging from 0 to 1. Unlike the Dice coefficient, a smaller Dice Loss value indicates higher similarity between the prediction and the ground truth.

3.5.1 Inference

At inference time, we fuse BN layer, 3 × 3 convolution and residual layer into a single large convolution in CNN branch, which simplifies and speeds up calculations. The final result output is obtained by fusing the results of the detection head and the segmentation head. The fusion rule operates as follows: when the predicted segmentation region contains fewer than three pixels—a condition that empirically corresponds to noise-level activations rather than genuine small targets—the detection head's probability score is used as a gating signal. The segmentation output is retained only if the detection probability exceeds 0.5. This heuristic is motivated by the observation that true small targets typically activate both detection and segmentation heads simultaneously, while isolated noise pixels tend to trigger only the segmentation head. In our pilot study, this rule reduced the false alarm rate from noise by approximately 15% while maintaining a detection rate above 98% for genuine targets.

4 Experimental analysis

4.1 Experimental data and experimental setup

Currently, there is a scarcity of open-source datasets for infrared small target segmentation. The MD_SIRST dataset, the first publicly available dataset for deep learning-based infrared small target detection, contains 10,000 training and 100 test images. However, it primarily consists of synthetic images, limiting its applicability to real-world scenarios. Additionally, the dataset exhibits inconsistent target sizes, mixing normal-sized infrared targets with small targets, which compromises its suitability for dedicated small target detection tasks. In contrast, the NUAA-SIRST dataset (Dai et al., 2021a) comprises 427 real-world images with high-quality annotations. While it offers superior data authenticity, its limited size restricts its utility for data-driven deep learning models, which typically require large-scale training data.

To address this limitation, we employ an aggressive data augmentation strategy to expand the NUAA-SIRST dataset:

• Geometric Transformations: generated 1,708 augmented samples via rotation, flipping, and scaling.
• Noise Injection: created 2,135 synthetic samples by adding Gaussian and salt-and-pepper noise.
• Illumination Adjustments: produced 854 variants through contrast and brightness modulation.

The final augmented dataset contains 5,124 images, split into training (70%, 3,587 images), validation (10%, 512 images), and test (20%, 1,025 images). Crucially, the test set is constructed exclusively from augmented samples, and all augmentations are applied independently per split to ensure complete separation–no augmented variant of any training image appears in the test set.

The computer CPU model for the experiment is Xeon Silver 4110. CPU frequency 2.1 GHz, the highest RWF 3 GHz, production process 14 nm, 8 cores, and 16 threads. The graphics card is a GeForce RTX 3090 with 24 GB of graphics memory and the system type is 64-bit Ubuntu 20.4. The deep learning framework used was PyTorch 1.9. The network is trained using the Adam optimizer with an initial learning rate of 1 × 10⁻⁴, batch size of eight, weight decay of 1 × 10⁻⁴, and a total of 10,000 iterations. A cosine annealing learning rate scheduler with 500-step warmup is employed. The random seed is fixed to 42. The detection threshold is set to 0.5. The weight pre-training parameters of the backbone network are the pre-training on the ImageNet dataset.

4.2 TCNNISTD main structure comparison experiment

We conduct comparative experiments on six different network models, RepLK(18_layer), RepLK(34_layer), RepLK(50_layer), RepLK(101_layer), RepLK(152_layer), and DeiT, respectively. Table 1 demonstrates the experimental results of different subject network structures under the augmented dataset.

Table 1. Experimental results of three different network structures.

The best values are indicated in bold.

As can be seen in Table 1, the proposed method TCNNISTD with RepLK+DeiT achieves the best results on all metrics, including Dice and mIoU. In comparison, RepLK is the next best and DeiT is the worst. This demonstrates that the TCNNISTD method is indeed able to better solve the long-range dependence problem in infrared small-target detection, thus improving the detection precision and accuracy.

A more systematic analysis reveals two reasons for Transformer underperformance alone. First, DeiT is pre-trained on ImageNet where semantics are conveyed through texture and shape; infrared small targets (often smaller than 10 × 10 pixels) lack such features, causing self-attention to attend to background noise. Second, the fixed patch size of 16 × 16 is too coarse for sub-10-pixel targets–the signal is either pooled into a single token or split across tokens, diluting its representation. When combined with the CNN branch, the CNN provides high-resolution local features that the Transformer can globally contextualize, overcoming both limitations.

Figure 5 illustrates the segmentation results for different network body structures. The red circle is the target to be detected, the blue circle indicates a missed target, and the yellow circle is a false alarm. Here, we selected CNN(50_layer) and Transformer and CNN(18_layer) + Transformer, which have the best metrics values among their type structures, for a more intuitive comparison. In Figure 5a, the segmentation output of both Transformer and CNN show missed detection, while the segmentation result of CNN+Transformer accurately detects the target. On the one hand, because the pixels of the target to be detected are too few, the CNN network has a strong local feature extraction capability, but the target information is lost after many times of downsampling. Transformer, on the other hand, focuses more on global information extraction and has a poor ability to extract local information features leading to loss of target information. On the contrary, TCNNISTD combines the advantages of CNN and Transformer, so it can accurately extract target information. On the other hand, in Figure 5c, except for CNN+Transformer which has no background clutter, the rest of the methods segmentation result maps have background clutter at the same position. It also proves that the performance of the CNN+Transformer algorithm is much better. In addition, the proposed TCNNISTD method not only achieves good detection performance but also performs well in terms of the amount of data required for training. TCNNISTD can train an end-to-end network model using only a few datasets. It is more suitable for the current situation where infrared small target detection datasets are scarce.

4.3 Large kernel convolutional comparison experiments

Since the input size of the ResNet Layer 2 network block is 56 × 56, the final feature maps of either a single large kernel convolution or multiple large kernel convolutions are adjusted to 56 × 56 to match the ResNet Layer 2 network block.

Table 2 shows the results of the experiments using only a single large convolutional verification, with the first layer being a large kernel convolution and the second layer having a convolutional step size of 2.

Table 2. Experimental results for a single large convolutional kernel.

The best values are indicated in bold.

Table 3 shows the results of multiple large convolutional verification experiments without convolutional operations with step size 2. Note that only the convolution operation is modified here, the rest of the modules are the same.

Table 3. Multiple large convolutional kernel experiments.

The data presented in Tables 2, 3 suggest that the most effective configuration is attained using an initial convolutional kernel measuring 11 × 11, followed by a secondary kernel of 3 × 3. At the beginning of the experiment, RepLKNet was directly applied to downsample the feature maps of 56 × 56 with multiple convolutional kernels, but from Table 3, it can be concluded that the effect of multiple large kernel convolution is not as good as that of small kernel convolution. Thus, the first layer of large kernel convolution and the second layer of small kernel convolution with a step size of 2 are used, and as can be seen from Table 3, the best results are obtained when the convolution kernel is 11 × 11. This is because the pictures of the RepLKNet experiment are not as extreme as the actual scenes of the IR small targets. Infrared small targets, some of which are less than nine pixels in size, will lose target information directly by using large kernel convolution multiple times. On the other hand, large kernel convolution works better than 7 × 7 when paired with 3 × 3 convolution kernels. This is because the large kernel convolution ensures the effective receptive field of the image, while the small kernel convolution also preserves the underlying features of the image, and the combination of the two can give better results.

4.4 Comparative experiments on attention mechanisms

Different attention mechanisms are experimented with for the CNN branch, Transformer branch, and CNN fusion Transformer. The network only modifies the attention mechanism, and the experimental results are listed in Table 4.

Table 4. Different attention experimental result

The best values are indicated in bold.

As can be seen from Table 4, the Multi-head-attention mechanism (MHAM) performs well in several domains, however, after adding the network, the effect becomes worse. Considering that multi-head attention can extract more features compared to single-head, multi-head implements an attention mechanism for multiple messages. Since small target detection involves limited feature information, excessive feature associations may cause the target to blend too much with the background, leading to incorrect segmentation. From Table 4, it can be intuitively seen that in the CNN branch, the channel can have better results by increasing the attention, while the use of spatial attention in the fusion of CNN and Transformer features can effectively improve the detection performance.

Figure 6 demonstrates the segmentation comparison of the attention free mechanism and the TCNNISTD method, where the red box indicates the target to be detected, the blue circle is the missed target. As can be seen from Figures 6c–e, for a small number of targets to be detected with high intensity, when, with or without the attention mechanism, it performs well. However, when the background is complex and the target is weak, as in Figure 6a, leakage is produced when there is no attention. On the other hand, when there are too many targets to be detected as in Figure 6b, the model with no attention, sees multiple dense targets as background, leading to missed detection. It is not difficult to conclude that the incorporation of a suitable attention mechanism enhances the model's ability to recognize the target to be detected, and at the same time, it can reduce the interference of background and noise.

4.5 Feature fusion experimental analysis

This section test the weight values corresponding to the four maps in the loss function. The feature fusion produces multiple maps, and each map can be used as the final output of the feature map. To verify which map is the optimal choice, we output single map1, map2, map3, and map4, respectively. In particular, one map weight in the loss function is set to 1, and others are set to 0. The output in Table 5 demonstrates that map3 is the best feature map in the four maps for small target detection.

Table 5. Feature map optimal experiment.

The best values are indicated in bold.

To verify whether incorporating map1, map2, and map4 into the loss function calculation improves model performance, we added these maps to the loss function calculation, respectively. The experimental results are shown in Table 6.

Table 6. Loss function hyperparameter modification experiment.

The best values are indicated in bold.

As can be seen in Table 6, the output of only map3 in the loss function is the smallest. Then, the loss output are improved with map1, map2, and map4 added, showing that multiple maps could improve the performance for infrared small target detection.

To select the most suitable weights for each feature map of map1, map2, map3, and map4, we simultaneously add all the maps to the loss function. Different weights are set for the four maps, experimental results are displayed in Table 7.

Table 7. Maps weight experiment.

The best values are indicated in bold.

Since the final output result is map3, the map3 is set to a larger weight. As can be seen from Table 7, the best indexes can be obtained when the weights are set 0.2, 0.2, 0.4, and 0.2, for map1, map2, map3, and map4, respectively.

4.6 Comparative analysis of different detection methods

To verify the effectiveness of the model proposed in this paper, some advanced methods in infrared small target segmentation detection, including the ACM (Dai et al., 2021a), ALCNet (Dai et al., 2021b), UIUNET (Wu et al., 2023) and DNANet (Li et al., 2023) are selected for experimental comparison. Additionally, Unet (Raza et al., 2022), which is currently widely used in image segmentation was also selected.

To ensure fair comparison, all baseline methods are re-trained on our augmented dataset with the same training/validation/test split (70/10/20), input size (224 × 224), and augmentation pipeline. Official open-source implementations with recommended hyperparameters are used wherever possible.

DNANet, on the other hand, has designed a Dense Nested Interaction Module (DNIM) for infrared small target detection to achieve progressive interaction between high-level and low-level features, and thus also has good metrics values.

Beyond detection accuracy, we also evaluate the computational efficiency of all methods in terms of model parameters, inference time, and throughput, as summarized in Table 8. Our TCNNISTD model achieves a competitive balance with 26.25M parameters, 4.13 ms inference time per 192 × 256 input, and 242.4 FPS on a single RTX 3090 GPU. While lightweight single-channel methods such as ALCNet (0.20 M params, 0.92 ms, and 1,089.2 FPS) and DNANet (0.93 M, 1.93 ms, and 518.0 FPS) achieve faster raw throughput, they operate on grayscale input, whereas TCNNISTD processes three input channels to accommodate both RGB and infrared modalities. Among three-channel methods, TCNNISTD substantially outperforms UIUNet (50.54 M params, 9.40 ms, and 106.4 FPS) in both speed and parameter efficiency while delivering superior Dice and mIoU scores. The structural reparameterization used in our large-kernel CNN branch further contributes to inference efficiency by merging the multi-branch training structure into a single compact convolution at deployment, reducing memory access overhead without sacrificing representational capacity. These results confirm that TCNNISTD satisfies real-time deployment requirements, achieving over 240 FPS on consumer hardware while maintaining state-of-the-art detection accuracy.

Table 8. Experimental index values and inference efficiency for different algorithms on a single RTX 3090 at 192 × 256 input resolution.

The best values are indicated in bold.

To provide a more intuitive comparison of the advantages and disadvantages of each algorithm, we selected six representative images and displayed the results.

Figure 7 illustrates the output of different algorithm segmentation. In the figure, the red circle represents the target to be detected, the blue circle indicates a missed target, and the yellow circle signifies background clutter. It can be observed in Figure 7 that Unet has missed detection in Figures 7d–f. It is because the original Unet is not optimized for the property that there are few pixels in the small target, and the target information disappears after many convolutions. For Figure 7f, only TCNNISTD detect all the three small targets, the next closest being DNAnet, with the rest of the methods having substantial misses. It is mainly because the three targets to be detected in Figure 7f are too close and disturbed by the background, which causes the algorithm to recognize the three very close targets as the background. It is a sign of the algorithm's weak filtering ability for background and noise. And DNANet can maintain the distribution of small infrared targets at depth through repetitive interactions in DNIM. The background information of small targets can be well-integrated and fully utilized through repeated fusion and enhancement, therefore, the difference between the background and the target can be better distinguished to detect segmentation. However, this method also only reaches the level of being able to recognize, and the detected target is not complete enough. The parallel multi-branch structure of the Transformer and CNN of TCNNISTD realizes the simultaneous capture of global dependencies and local information and also introduces the attention mechanism, so it has a stronger ability to recognize the background and target. This is complemented by the larger effective receptive field given by the large kernel convolution so that the target to be detected can be more completely segmented.

We also conducted comparative experiments on three public infrared small-target detection datasets—NUDT-SIRST, SIRST-aug, and AntiUAV OTSU—to evaluate the generalization capability of different networks.

4.7 Cross-dataset evaluation

To further assess the generalization ability of our method, we evaluate on multiple public benchmark datasets with diverse characteristics.

The NUDT-SIRST dataset contains 1,327 real-world infrared images captured from various scenes, including urban, suburban, and maritime environments, with high-quality pixel-level annotations for small targets under different background conditions. The SIRST-aug dataset is the augmented version of NUAA-SIRST described in Section 4.1, containing 5,124 images with diverse augmentation strategies (geometric transformations, noise injection, and illumination adjustments), with all splits kept strictly separated to prevent data leakage. The AntiUAV OTSU dataset is derived from the Anti-UAV challenge Jiang et al. (2021), consisting of 2,050 infrared images captured by drone-mounted cameras with OTSU-binarized pixel-level annotations. The images, originally 16-bit PNGs at 640 × 512 resolution, are normalized to 8-bit for network processing, divided into 1,450 training and 600 testing images covering diverse drone flight scenarios including different altitudes, background complexities (sky, building, tree, and mountain), and lighting conditions (daylight, dusk). Unlike traditional infrared datasets, AntiUAV OTSU presents unique challenges: extreme viewpoint variation from drone motion, diverse backgrounds, and a high proportion of sub-10-pixel targets with extremely low signal-to-clutter ratios, making it a challenging benchmark for drone-based infrared small target detection.

Table 9 presents the quantitative comparison with state-of-the-art methods on these three public datasets.

Table 9. Quantitative comparisons (Dice/mIoU) with state-of-the-art models on three public datasets.

The best values are indicated in bold.

From Table 9, several observations can be made. DNANet could not obtain satisfactory performance on the AntiUAV OTSU dataset (0.6407 Dice, 0.5046 mIoU). In contrast, TCNNISTD maintains consistent performance across all three datasets, ranking competitively on each. On the AntiUAV OTSU dataset specifically, UIU-Net achieves the highest overall Dice (0.7145) and mIoU (0.6612), followed by ISTDU-Net (0.6953 Dice) and TCNNISTD (0.6686 Dice). TCNNISTD achieves the best results on SIRST-aug (0.7928 Dice, 0.6698 mIoU) and demonstrates robust generalization across different data distributions. The competitive performance of TCNNISTD on drone-captured imagery confirms the effectiveness of the parallel dual-branch design for handling the high variability in background and target appearance typical of drone scenarios. The relatively lower performance of all methods compared to the augmented NUAA-SIRST results (Table 8) underscores the additional difficulty posed by real-world drone data, including camera motion blur, rapidly changing backgrounds, and extremely faint targets. These results highlight the need for further research on domain adaptation and robust feature learning for drone-based infrared small target detection.

To further qualitatively evaluate the detection capabilities of different methods, Figure 8 presents visual comparisons of four representative methods (ISTDU-Net, UIU-Net, DNANet, and TCNNISTD) on four AntiUAV OTSU test images with varying target sizes (14px, 41px, 15px, and 203px from top to bottom). The results demonstrate that TCNNISTD consistently achieves competitive detection performance, particularly on small targets. On the tiny-target sample (Row 1, 14px), TCNNISTD obtains the highest Dice score of 0.9041, significantly outperforming ISTDU-Net (0.8276), UIU-Net (0.8136), and DNANet (0.0000). For the medium-small sample (Row 3, 15px), TCNNISTD achieves the best Dice of 0.9583, confirming its effectiveness on sub-20-pixel targets. On the large-target sample (Row 4, 203px), ISTDU-Net slightly edges ahead with 0.9877 Dice, while TCNNISTD maintains competitive performance (0.9435). Notably, DNANet fails to detect targets in three out of four samples, resulting in Dice scores of 0.0000. These visual results corroborate the quantitative findings from Table 9, where TCNNISTD consistently delivers robust detection across a wide range of target scales. The ability of TCNNISTD to maintain high detection accuracy on extremely small targets (sub-20px) can be attributed to its parallel CNN-Transformer architecture, which simultaneously captures local details and global context, while the large-kernel convolution branch preserves spatial resolution for fine-grained target discrimination.

5 Conclusion

To improve the detection performance of infrared small targets in complex backgrounds, this paper proposes a CNN and Transformer parallel network named TCNNISTD. It combines the advantage of CNN to extract detailed information about the target region and the ability of the Transformer to extract global information in the whole image. Many existing networks employed multiple smaller convolutional kernels for feature extraction to prevent target information loss. In contrast, we introduce large-kernel convolution to increase the effective receptive field of the model, which is especially beneficial for capturing the subtle context around tiny targets. Moreover, channel attention and spatial mechanisms are incorporated into our model to amplify the significance of small targets while suppressing background noise. By diminishing noise interference, the model can focus more on critical feature areas, thereby enhancing its capacity for accurate segmentation and detection. Additionally, we design a dedicated feature map fusion module for the CNN and Transformer branches and modify the loss function to fully leverage multi-level features. Experimental results validate that compared with other benchmark models, the proposed TCNNISTD demonstrates competitive infrared small target detection performance with enhanced robustness.

Although, we have proposed a parallel structure of Transformer and CNN for infrared small target detection, the employed Transformer module is a basic one and lacks the specific improvements made to the CNN branch. In the future, we will design a more suitable Transformer branch tailored for infrared small target detection, especially for scenarios that demand high sensitivity and low false alarm rates. Furthermore, the model parameters in the proposed network are relatively large. Reducing the model scale while maintaining detection accuracy is another important research direction for deploying the method on resource-constrained edge computing systems.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Kaleem Siddiqi, McGill University, Canada

Reviewed by: Laixiang Xu, Hainan University, China

Honggui Cao, Kunming University of Science and Technology, China