A deep learning-based decision fusion framework for image forgery localization

1 Introduction

We live in an era where digital media are completely intertwined with our life. From entertainment to social interactions, from news to crime investigations, digital images play a role in virtually every aspect of our existence. As an obvious consequence, the demand for being able to prove the source and authenticity of images has increased dramatically in the last years, and the ease of creating credible hoaxes further pushed such demand.

At first, the scientific community responded to such a demand by proposing active authentication techniques, such as digital watermarking (Cox et al., 2007). While being successful in selected areas, such active technologies failed to reach the general public, thus calling for the development of different solutions. Multimedia Forensics has thus emerged as the branch devoted to the passive investigation of multimedia contents, meaning that an analyst has no control over their generation process. The core idea is that each step in a media's generation chain (optical system, color filter array demosaicking, sensor, compression, internal processing, etc.) leaves some traces in its content or metadata. Thus, researchers developed mathematical models for such traces and built hundreds of different “model-based algorithms” for detecting anomalies or inconsistencies, allowing to reveal the presence of possible manipulations (Piva, 2013; Kakar et al., 2011; Feng et al., 2012; Qiao et al., 2018).

Despite the abundance of solutions, authenticity analysis remained a challenging task, which requires an expert's work. One of the main issues is that each model-based algorithm typically leverages on a single trace, and its applicability is therefore limited to scenarios where such trace is present and sufficiently preserved to be detected. Therefore, a thorough analysis needs to consider the outputs of several complementary algorithms and jointly interpret them. However, such interpretation is non-trivial: the relationships between different traces are often unclear, and the reliability of algorithms may depend on one or more factors such as compression quality, resolution, presence of saturation, etc. On top of that, analysts are in some situations overwhelmed by massive amounts of images whose authenticity needs to be checked promptly (think, for example, to pictures of documents submitted to home offices).

In an attempt to facilitate the expert's work, researchers have investigated decision fusion techniques (Liu et al., 2021), specifically tailored to the image forgery detection (Fontani et al., 2013b; Barni et al., 2012) and localization (Ferrara et al., 2015; Mareen et al., 2023) tasks. These fusion systems were shown to improve performance compared to every single algorithm alone. However, they still required a fair amount of work in order to understand: i) the relationships between the forensic traces exploited by each algorithm, which had to be provided by the expert, and ii) understanding which are the reliability factors that influence each algorithm's performance, and provide suitable reference material needed for mapping such factors into reliability scores.

More recently, the relevant developments of machine learning techniques, especially in the field of deep neural networks, found their way into Multimedia Forensics (Yang et al., 2020). Deep learning-based image forensics systems (Zhang et al., 2022; Wang et al., 2022; Lou et al., 2025; Liu Y. et al., 2025; Kong et al., 2025) often yield impressive performance in the experimental validation carried out by their inventors, but their reliability rapidly drops as the difference between images used for training and those used for testing increases. This fact became harshly evident in a recent challenge for deepfake video detection: the best performing method used a deep network trained on part of the available dataset, and achieved an Area Under the Curve (AUC) of 0.984 on the test dataset, which, however, dropped to 0.717 when tested on real videos (Dolhansky et al., 2020). Moreover, while deep learning-based methods are undoubtedly valid for some authentication scenarios (e.g., to be used by a social media platform to flag fake contents automatically), they are far less suitable in the forensic scenario, where explainability, interpretability, and related concepts (Barredo Arrieta et al., 2020) are of the utmost importance.

We can thus say that, currently, the state of the art of Multimedia Forensics presents us with:

• a large variety of model-based image forensic algorithms, which are typically effective only for detecting manipulations that alter a specific trace, and whose reliability varies strongly depending on the image content;
• several decision fusion approaches, that would benefit from more advanced methods for understanding trace relationships and content-based reliability;
• promising results of deep-learning-based methods, which, however, tend to fail in generalizing to unseen examples.

Motivated by the three limitations outlined above, the objective of this work is to design a forgery localization framework that synergistically combines the interpretability and controlled behavior of model-based forensic tools with the learning capabilities of deep neural networks, while overcoming the generalization bottlenecks of purely data-driven approaches. Specifically, we address the gap that existing fusion methods rely on manually defined relationships between forensic traces and heuristic reliability models, which limits their scalability and adaptability to new manipulation types. At the same time, we tackle the generalization gap of deep learning methods, which excel on in-distribution test data but fail under covariate shifts. Our central motivation is to shift the learning target from pixel-level image statistics, which are highly content-dependent and domain-sensitive, to the relationships among multiple model-based forensic maps, which we hypothesize to be more stable across synthetic and real-world forgeries. By doing so, we aim to bridge the gap between classical multimedia forensics and modern deep learning, offering a solution that is both effective and explainable.

Based on this motivation, this paper addresses two specific research questions. RQ1: Can a deep neural network, trained exclusively on synthetically generated forgeries, learn to accurately fuse multiple model-based forensic localization maps when applied to real-world manipulated images? RQ2: What are the key criteria for generating synthetic training data that ensure the learned fusion rules generalize across different manipulation operations. Answering these questions allows us to validate the core hypothesis that relationship learning among forensic traces is a more generalizable strategy than content-based learning for image forgery localization.

In our opinion, it is time to synergistically exploit these existing results, as wished by the author of a recent overview (Verdoliva, 2020). In this work, we propose a framework that uses deep learning to fuse forgery localization maps provided by model-based image forensics tools. Our approach aggregates the outputs of multiple model-based tools in a single multichannel map. A deep neural network based on the U-Net architecture (Ronneberger et al., 2015) is then used to produce a fused forgery localization map.

Most importantly, we show that the fusion system can be trained using only synthetically generated data and applied to real-world images with a minimal performance loss. This improved generalization capability is the result of asking the network to learn the relationships between forgery localization maps provided by different tools: such relationships are associated with the traces analyzed by the algorithms, not to the analyzed image content, and so they transfer much more effectively from synthetic to actual forgeries.

We believe that our approach combines the strengths of existing results since it allows us to:

• aid the expert in jointly interpreting several model-based algorithms output, leveraging the capabilities of deep neural networks to learn complex relationships from data;
• increase the generalization capability of deep learning for Multimedia Forensics compared to state-of-the-art methods based on networks directly applied to image pixels.

Our contributions mainly focus on two aspects. First, to enhance the performance of fusion-based approaches for forgery image localization, we propose a deep learning framework that fuses localization maps generated by multiple forensic tools. The framework automatically learns the complex relationships among different forensic traces to improve localization accuracy. Second, we introduce important criteria for generating synthetic forgeries, aiming to ensure the generalization capability of the proposed framework across diverse manipulation scenarios.

The rest of this paper is organized as follows. In Section 2, a brief review of decision fusion methods for image forgery localization is given. In Section 3 the proposed algorithm is described. The results of the experimental analysis carried out to evaluate the performance of the proposed method are discussed in Section 4, whereas in Section 5, some conclusions are drawn.

2 Related work

2.1 Single forensic tool

Much of the existing research in image forensics has been devoted to the output of a single forensic tool. In this paradigm, a dedicated tampering detection framework is designed to produce a single tampering probability map from the input image. These approaches generally fall into two categories: model-based methods and deep learning-based methods. Classical model-based algorithms detect manipulations by analyzing specific traces left during image acquisition and processing. These include methods for detecting JPEG compression (Bianchi and Piva, 2012; Choudhary et al., 2024), CFA interpolation artifacts (Popescu and Farid, 2005; Ferrara et al., 2012), and image splicing (Cozzolino et al., 2015a).

More recently, several learning-enhanced approaches have been proposed. For instance, DenseFCN (Zhuang et al., 2021) employs an encoder-decoder architecture for image tampering localization, integrating dense connections and dilated convolutions to enhance localization performance. MVSS-Net (Dong et al., 2023) jointly exploits the noise view and explicitly extracted boundary artifacts to learn manipulation detection features. TruFor (Guillaro et al., 2023) combines camera model fingerprints with visual artifacts, achieving state-of-the-art performance by leveraging all-round forensic clues. CFL-Net (Niloy et al., 2023) enhances the standard cross-entropy loss by incorporating a contrastive loss component to achieve a more general approach. VGG-UNet (Choudhary et al., 2024) integrates a CNN architecture with error level analysis (ELA) for image forgery localization. DFST-UNet (Yang et al., 2025) integrates Swin Transformer blocks into the U-Net architecture to capture long-range context information and identify forged regions at different scales. Additionally, a dual-stream encoder is proposed to comprehensively expose forgery traces in both the RGB domain and the frequency domain. FMG-Locator (Liu J. et al., 2025) addresses the specific challenge of multi-person scene images using facial mask guidance and dual attention mechanisms. Pixel-Inconsistency Modeling (Kong et al., 2025) is a learning-based approach that captures inherent pixel inconsistencies in forged images by integrating global and local pixel inconsistency features extracted from two streams.

Table 1 presents a comparison of state-of-the-art deep learning-based image forgery localization methods alongside our proposed approach. As shown in the table, the majority of recent deep learning methods operate directly on image pixels and consequently experience generalization degradation when applied to out-of-distribution data. In contrast, our method deliberately operates on forensic tool outputs, learning relationships among forensic traces rather than pixel-level statistics, thereby achieving strong generalization even when trained exclusively on synthetic data.

Table 1. Comparison of state-of-the-art deep learning-based image forgery localization methods.

The bold text highlighted the key differences between the proposed method and existing SOTA methods.

2.2 Decision fusion for multimedia forensics

The research activity concerning the fusion of multimedia forensic tools is somewhat limited, but it demonstrates that information fusion outperforms single tools, both in controlled scenarios, as well as in several multimedia forensics competitions (Verdoliva, 2020). In general, we can distinguish between three different approaches to fusion:

• Fusion at the feature level, where the features used by different methods are combined, possibly reduced, and employed to train an aggregated system;
• Fusion at the score level, where the soft-valued decisions or maps produced by various methods are combined together;
• Fusion at the abstract level, where the binary decisions or maps produced by various methods are combined.

Back in 2006, Bayram et al. (2006) tested different forensic features in the presence of single and multiple manipulations. Their experimental results showed that in the case of a single manipulation, it is best to select features from the general pool of all categories and then apply a fusion at the feature level. In the presence of multiple manipulations, instead, the best solution was to exploit different types of classifiers, one per manipulation, and then directly fuse their decisions. After that, several techniques have been proposed to fuse the information at the feature level (Hsu and Chang, 2008; Zhang and Kong, 2009), and recently, a fusion system has been proposed (Walia et al., 2021) which constructs a 1160-dimensional feature vector by combining features extracted from the image luminance using a neural network, and other Markov-based features extracted from the quaternion discrete cosine transform of the image. Other approaches work at the score level (Fontani et al., 2011, 2013b; Barni et al., 2012); in particular, (Fontani et al. 2011, 2013b) proposed a framework based on the Dempster-Shafer theory (DST) (Dempster, 1967) of evidence to fuse outputs at the score level. The framework allowed to exploit the knowledge about the reliability of tools and compatibility between different traces of tampering; experiments showed superior performance compared to all the considered forensic tools and also to abstract-level fusion and support vector machine (SVM)-based fusion. In Barni et al. (2012), a decision fusion framework based on the Fuzzy Theory was proposed; by relying on the known ability of such theory to deal with inaccurate and incomplete information, the model demonstrated its superiority with respect to computing the “Logical OR” of the outputs produced by five forensic tools in a scenario of cut&paste tampering. It has to be noted, however, that the mentioned frameworks (Fontani et al., 2011, 2013b; Barni et al., 2012) allowed fusion of detection scores, not forgery localization maps: the analyst had to select a region of the image and a fused decision was produced for the selected region. In a later work (Fontani et al., 2013a), it was shown that the overall performance of the above frameworks could be further improved by taking into account background information, such as local properties of analyzed images. Moreover, the fusion can also include tools for anti-forensics, as shown in Fontani et al. (2014).

Ferreira et al. (2016) modeled the combination of different methods on a multi-scale and multi-directional Behavior Knowledge Space (BKS) (Huang and Suen, 1995) classification for the detection of copy-move forgeries. The method outperforms the common BKS approach and the classification of each method used in the fusion, but it has the drawback that it fails when there is no complementarity of the underlying classifiers for a given image. In Shelke and Prasad (2019), a new decision fusion method for the detection of tampering in JPEG images was provided. The algorithm, defined Dragonfly Bayes Fusion System, was developed by integrating the Naive Bayes classifier with the Dragonfly optimization algorithm; the outputs separately obtained by six forensic tools were fused for generating the optimal decision. The algorithm was tested on a limited set of JPEG images and compared against existing methods, such as Fuzzy Theory based classification, rule-based classification, average method, and weighted average method, showing improved performance.

As to forgery localization, a simple pixel-level fusion of different forensic tool outputs, based on complementary hypotheses, has been investigated in Cozzolino et al. (2013a,b); Gaborini et al. (2014): careful fusion of detector's output largely outperforms individual detectors, but the methods do not consider tool's reliability and compatibility, and they are based on simple rules for fusion at the score level, like sum, product, or logical disjunction/conjunction at the abstract level. In (Cozzolino et al., 2014) an image forgery localization technique was proposed that fuses the outputs of three complementary tools, based on sensor noise, local descriptors, and block-matching (for copy-move forgeries). The fusion is carried out through a map combination strategy based on the reliability associated with each binary mask. The technique ranked first in phase 2 of the first Image Forensics Challenge organized in 2013 by the IEEE Information Forensics and Security Technical Committee (IFS-TC). Recently, a neural network-based fusion method working at the feature level has been presented in Liu and Pun (2020), where authors combine features targeted to reveal JPEG-based inconsistencies with features targeted to reveal noise inconsistencies. Authors suggest that using low-level features should increase the generalization capability of the network, and results in a realistic tampering database (Korus and Huang, 2016a) seem to verify this hypothesis, although the performance obtained by the network is only slightly better than that of a Color Filter Array-based tool presented back in 2012 (Ferrara et al., 2012), which proves to be rather effective in that specific dataset. A more modern interpretation of fusion at the feature level is proposed by Liu and Pun (2018): inspired by the fact that in deep learning the selection of features is often left to a neural network, they extract layers from several networks that were previously trained to localize forgeries based on different traces, and then combine such layers using another network. In Ferrara et al. (2015), DST was again used to define an algorithm for unsupervised multi-clue forgery localization, which merges information provided by a set of forensic tools with semantic-based information. Relying on the properties of those localization tools, based on mixture models, and by introducing global variables, which embed global map features into local decisions, the performance is considerably improved compared to that of single tools. DST has found further applications in two recent works (Ho and Retraint, 2019; Tang and Zhang, 2018); however, the method in Ho and Retraint (2019) assigns all the belief mass to singletons, thus voiding the difference between DST and classical probability theory and turning the combination rule into a surrogate of a product combination, while the method in Tang and Zhang (2018) simply applies the same theoretical framework of Ferrara et al. (2015) to a different set of algorithms. In Korus and Huang (2016a), instead, contextual information is taken into account by means of Conditional Random Fields to model dependencies between neighboring image patches in a content-adaptive way, in such a way to increase the localization of complex shapes, while in (]Korus and Huang (2016b) and Korus and Huang (2017) multi-scale fusion is studied to improve the localization accuracy of tools operating in a sliding-window modality. In particular, three fusion methods, based on energy minimization, iterative top-down and bottom-up fusion are designed in the context of JPEG splicing forgery localization. The proposed strategies show improved performance compared to four reference methods, including decision-level fusion by majority voting, measurement-level fusion by candidate map averaging, supervised learning with support vector machines, and K-means clustering. However, the method is not designed to fuse maps provided by different tools, it is rather a way to increase the reliability of a single tool. In Li et al. (2017), two tampering localization methods are fused at the score level by manually designing a decision curve with two weight parameters based on the observation of the normalized distributions of pristine pixels and fake pixels on a set of training images.

Tests show that the proposed framework achieved the highest F1-score in the IFS-TC Image Forensics Challenge. Recently, authors of Yao et al. (2020) proposed a fusion system that combines a CNN-based forgery localization method with a reliability map, which is expected to identify local regions of an image where forgery localization is generally less reliable (e.g., highly textured pixels). The method is shown to achieve a very high localization accuracy, but all images used in the validation were obtained from the Dresden dataset (Gloe and Böhme, 2010), and this fact leaves the generalization capability of the algorithm unexplored. Moreover, although it contains a fusion step, the method in Yao et al. (2020) is not conceived to merge the outputs from different tools, but rather to improve the output of one single detector proposed by the authors.

3 Proposed method

In the proposed method, we fuse at the score level the output maps produced by several existing forgery localization tools, as visually depicted in Figure 1. In order to perform this operation, we need to build a neural network architecture capable of receiving multiple images (representing the original localization maps) as inputs and producing a single image (representing the “fused” map) as output. We also need to define how to train such a network in a way that favors its generalization capabilities. These two points are discussed separately in the following. We remark that, as Figure 1 shows, the proposed system does not take the original image pixels as input. This design choice is intentional, as directly using image pixels could increase the risk of overfitting to the training data. Instead, our system focuses solely on learning the relationships among the localization maps generated by different forensic tools. By decoupling from the raw image content, the model is better positioned to generalize across diverse and unseen examples. We also stress that, although we provide a realization of the proposed system in Section 4.6, the designed architecture and strategy can be applied to an arbitrary set of localization algorithms.

3.1 Network architecture

Commonly used neural network architectures for image processing are based on convolutional layers, where multiple filters are used to process the information contained in the inputs by exploiting spatial correlation. Those architectures, however, are usually designed for classification and regression tasks, where a single discrete or continuous value is obtained by stacking multiple convolutional and pooling layers followed by a fully-connected component.

We propose to use a different approach inspired by U-Net (Ronneberger et al., 2015), a fully convolutional network. More specifically, our architecture is composed of a contracting and an expanding parts. Both these parts are composed of multiple blocks. Contracting blocks aim to compress the information contained in their inputs by performing two 3 × 3 convolutions followed by a 2 × 2 max-pooling. As the number of filters in each contracting block is larger than the block's input channels, those operations result in a larger number of lower resolution feature maps. The first contracting block is presented several localization maps as multiple channels of a single image. The output of the contracting part is then fed to a central block composed of two 3 × 3 convolutional layers, whose result is in turn used as input to a series of expanding blocks in order to rebuild a full-resolution fused map. Each expanding block firstly upsamples the input map, doubling each dimension, and then performs one 3 × 3 convolution on that. Then, to rebuild the missing spatial information, the output of the convolution is concatenated filter-wise to the full-resolution output of the corresponding contracting block (i.e. the maps just before the max-pooling operation). Finally, this combined map is processed by two 3 × 3 convolutions. The output of the expanding part is finally processed by one 3 × 3 and one 1 × 1 convolutions to build the output map. In order to regularize the network, we used two dropout layers, each with a dropout probability set to 0.5, one just before the max-pooling layer of the last contracting block, and one after the central block. We report details and a pictorial representation of the proposed architecture in Table 2 and Figure 2, respectively.

Table 2. The proposed network architecture.

The Dropout* layer is used only in the last contractive block.

Preliminary results on the task at hand suggested using three contracting blocks and three corresponding expanding blocks, with a number of filters of [16, 32, 64] and [64, 32, 16], respectively, and a central block with 128 filters.

It is worth clarifying how the multiple forensic maps are embedded as input to the network. The proposed framework does not employ a separate embedding module. Rather, the output maps from each of the N forensic tools are individually preprocessed and resized to 512 × 512 pixels, then stacked along the channel dimension to form a single N×512 × 512 tensor. This tensor serves as the direct input to the first convolutional layer of the U-Net. The first contracting block, with 16 filters of size 3 × 3, convolves across the channel dimension, thereby learning a weighted combination of the input maps at each spatial location. Subsequent convolutional layers progressively learn higher-level embeddings of the fused information. Thus, the input maps are embedded simply through channel-wise stacking, and the network learns the optimal fusion strategy end-to-end without any manually designed embedding function.

3.2 Training strategy

We now aim to define a general strategy to create suitable training images for the proposed fusion network. Indeed, although many image forensics datasets exist that provide realistic forgeries, the total number of available images remains limited. Moreover, most datasets contain images that have been forged with the same process, which may not be compatible with any of the algorithms that we need to fuse. On the contrary, there is a vast abundance of original images, e.g., in datasets such as Shullani et al. (2017); Gloe and Böhme (2010); Galdi et al. (2019); Dang-Nguyen et al. (2015). We thus propose to use original images to create synthetic forgeries to train the network. For each forgery localization algorithm that has to be fused, we propose to generate synthetic forgeries following these criteria:

• gather a sufficiently large dataset of original images, possibly spanning a heterogeneous set of acquisition devices, storage formats, depicted content;
• create synthetic forgeries respecting the working assumptions of the algorithm. A general approach could be to follow the same procedure adopted by the authors of the algorithm in their experimental validation;
• during forgery creation, perform data augmentation spanning those properties known to influence the algorithm's reliability. For example, saving forgeries at different compression qualities would be recommended for algorithms based on compression traces.

Step 2 ensures that, for each algorithm to be fused, some forgeries exist in the training dataset where such an algorithm has chances to provide a meaningful output. The other two steps aim at providing the network with training data that explores the reliability of each algorithm depending on some known influencing factors (Step 3) and on different pixel content (Step 1). It is worth noticing that a certain kind of synthetic forgeries could be compatible and useful for several algorithms.

The formal algorithmic description about training and inference of the proposed fusion network are provided as shown in Algorithms 1 and 2.

Algorithm 1 Training of the proposed fusion network. Screenshot of pseudocode outlining a neural network training pipeline for tampering detection, including initialization of datasets, data augmentation, input preparation, U-Net model training, validation, early stopping, and saving optimal model parameters for a forensic image analysis task.

Algorithm 2 Inference (forgery localization using trained fusion network). Screenshot of an algorithm presented as formatted pseudocode for generating a fused forgery localization map using multiple forensic tools, normalization, resizing, stacking, neural network fusion, and optional binary thresholding steps.

4 Experimental validation

We performed a series of experiments to answer two questions: (1) whether the proposed method is capable of combining the information contained in multiple localization maps in a meaningful way; and (2) whether the proposed method is useful in a real-world scenario where tampered data distribution may be different from the one used for training. In order to answer these questions, we tested our approach in two different scenarios.

4.1 Match scenario

In the first scenario, we evaluated the performance of our network on several well-known image forensics datasets. We split each dataset in two disjoint subsets, one of which was used to train the network while the other was used to evaluate the performance. This experimental protocol is commonly used to estimate an approach's performance on unseen data. It should be noted, however, that this estimation may be inconsistent with real world scenarios where images may come from completely different distributions.

4.2 Mismatch scenario

In the second scenario, images from well-known image forensics datasets were only used at test time, while data from a new, synthetic dataset was used to train our network. By doing so, we obtain a more realistic estimation of the performance that can be reached by our approach when fusing localization maps from images exhibiting completely different characteristics from the ones used during training.

4.3 Image forgery localization tools

For our experiments, we selected three forensic tools capable of localizing tampered regions, each one exploiting different inconsistencies introduced by a manipulation: CFA (Ferrara et al., 2012), SpliceBuster (Cozzolino et al., 2015b), and ADJPG (Bianchi and Piva, 2012) as shown in Equation 1.

In Ferrara et al. (2012), a forgery localization method is proposed by exploiting the fact that a manipulation will corrupt a periodic correlation pattern in the pixel domain introduced by the color filter array (CFA). This filter, with a periodic pattern, is used in such a way that each sensor element receives light only in a range of wavelengths corresponding to one of the three primary colors (i.e., red, green, blue); the missing color amplitudes are then obtained by means of an interpolation process using the neighboring pixels, an operation known as demosaicing, whose result is the introduction of a periodic pixel correlation. Starting from such an assumption, the authors propose a new feature, cast in a Bayesian framework, that measures the presence/absence of these artifacts even at the smallest 2 × 2 block level, thus providing as final output a forgery map indicating with fine localization the probability of the image to be manipulated.

In Cozzolino et al. (2015b) noise artifacts introduced by the whole acquisition process are exploited since possible anomalies can reveal a presence of tampering. In particular, the local co-occurrence map of high-pass noise residual of the image is computed, and its local statistical properties are studied through an expectation-maximization algorithm that clusters these features and reveal inconsistencies.

In Bianchi and Piva (2012), a forensic algorithm able to discriminate between original and forged regions in JPEG images, under the hypothesis that the tampered image presents a double aligned JPEG compression(A-DJPG) is presented. Based on an improved statistical model characterizing the artifacts of DCT coefficients that appear in the presence of A-DJPG, the proposed algorithm, following a Bayesian approach, automatically computes a likelihood map indicating the probability for each 8 × 8 DCT block of being doubly compressed.

We ran each tool on every image contained in our datasets, and we collected the output localization maps to be used as inputs of our fusion network. It should be noted that the outputs of the forensic tools are floating point maps where pixels in the tampered and pristine areas are characterized by different magnitudes. However, we cannot know a priori whether low output values are associated with pristine pixels or vice versa. This is an inherent ambiguity that could jeopardize the learning process of our network. In order to partially solve this problem, we post-process each localization map by firstly normalizing it to [0, 1]. Specifically, for the CFA-based method, we applied the exponential function exp(·) to its output, as it is originally in the logarithmic scale. For the SPBST method, we normalized the output using min-max normalization:

Normalized=output-minmax-min(1)

For the ADJPG method, we computed the normalized value using the sigmoid-like function as shown in Equation 2:

Normalized=11+exp(output)(2)

And then by using a heuristic to guess which end of the spectrum represents the tampered area. We indeed assume that the pristine area is always larger than the tampered one; thus, we assign the tampered class to the value associated with the minority of pixels in the map. We thus possibly flip the value of each pixel so that higher values always correspond to a higher tampering probability. We also resize the maps to 512 × 512 pixels to allow the network to process uniformly sized maps.

4.4 Datasets

In the following section, we will provide a description of each dataset used in the experimental validation of the proposed method. To provide a clear overview of the data used in our experiments, Table 3 reports the size of training, validation, and test subsets for each dataset. Below we summarize the key characteristics and sizes of each dataset.

Table 3. Size of training, validation, and test subsets for each dataset.

4.4.1 IEEE IFS challenge

This dataset was initially introduced in the Image Forensics Challenge¹ organized by the IEEE Information Forensics and Security Technical Committee (IFS-TC). The collection consists of 2200 images in PNG format: 1050 pristine images, 1150 tampered images, and ground truth masks. The manipulation techniques used in the dataset creation include copy-pasting and splicing.

4.4.2 Korus realistic tampering dataset

This dataset (Korus and Huang, 2016b,a)² consists of 440 images in TIFF format: 220 pristine, 220 manipulated, plus ground truth masks. The tampering techniques include object insertion and removal, performed by hand with well-known editing software such as GIMP and Affinity Photo.

4.4.3 FaceSwaps

This dataset, also known as UMD Face Swap Dataset (Zhou et al., 2017)³, consists of an extensive collection of automatic face tampering performed by the iOS app SwapMe⁴ and the open-source software FaceSwap.⁵ After selecting a face pair (target-source) each application performs the manipulation swap. A final post-processing of boundary-blurring, resizing, and blending is added to camouflage possible imperfections due to the automation process. The collection includes 2010 tampered images in JPEG format: 1005 tampered by SwapMe and 1005 by FaceSwap. In addition, we built the ground truth mask for each manipulation according to the location information of the tampered face.

4.4.4 IMD2020

This dataset, also known as IMD2020 Real-Life Manipulated Images (Novozamsky et al., 2020),⁶ consists of 2010 real-life manipulated images downloaded from the Internet. The collection includes the 2010 pristine images and the ground truth masks manually generated to identify the tampering.

4.4.5 Synthetic dataset

A novel automatic tampering dataset to test the proposed approach. It uses 500 raw images in TIFF format captured by digital still cameras (Canon 5D, Canon EOS 450D, Lumix G2, Nikon D50, Nikon D7000), and 3500 JPEG images crawled from Flickr. The raw contents were tampered via splicing, as shown in Figure 3; according to the criteria defined in Section 3.2, different tampering procedures have been designed in order to test consistently the CFA (Ferrara et al., 2012), SpliceBuster (Cozzolino et al., 2015b), and ADJPG (Bianchi and Piva, 2012) forensics tools. The workflow of the manipulation consists of randomly replacing one area of an image with another one that has been rotated (with an interval of [−10, 10] degrees), resized (by a scale factor in the interval ([0.75, 1.25]), and JPEG compressed with a quality factor in the range {70, 75, 80, 85, 90, 95, 100}. Overall, the Synthetic dataset contains 3200 tampered images and their corresponding ground truth masks. Some examples of synthetic images are shown in Figure 4.

Finally, we note that ADJPG can obviously only work on JPEG images. Examples from the IEEE IFS-TC Challenge and Korus Realistic Tampering datasets, however, are in PNG format and are thus unsuitable to this kind of analysis. When building ADJPG maps for these datasets we thus use a dummy localization map containing 0.5 for each pixel.

4.5 Performance metrics

We evaluated the performance obtained by our approach using two different performance metrics: area under the ROC curve and intersection over union.

The Area Under the Receiver Operating Characteristic (AUROC) curve (Hanley and McNeil, 1982) is a commonly used performance metric for binary classifiers that produce soft outputs. In those systems, choosing different discriminating thresholds would impact the ability to correctly separate positive from negative samples. The AUROC curve represents all the possible trade-offs between high true positive rates and low false negative rates. The curve is designed in such a way that the higher areas under it, the more accurate the classifier is. For the purposes of our experiments, we consider each pixel as a sample: all the tampered pixels correctly identified as such represent the true positives, while all the pristine pixels wrongly classified as tampered represent the false positives.

Intersection over Union (IoU) is a metric commonly used in localization problems to evaluate how well the output of a method overlaps with the ground truth. More specifically, the IoU is given by the ratio between the number of pixels correctly classified as tampered (intersection), and the number of pixels that have been either correctly classified as tampered, wrongly classified as pristine, or wrongly classified as tampered (union). In this way, the score will penalize both methods that only localize a fraction of the tampered area (as the intersection will be small), and methods that wrongly identify large parts of the image as tampered (as the union will be large). In order to evaluate the Intersection over Union score, continuous maps containing soft decisions had to be turned in binary maps. To do so, we used the method proposed by Otsu (1979) to obtain the optimal decision threshold.

4.6 Experimental setup

We implemented the method as described in Section 3 in Python using PyTorch (Paszke et al., 2019). We trained our network using the Adam (Kingma and Ba, 2017) optimizer with a learning rate of 10⁻⁴, minimizing a loss consisting of a pixel-wise binary cross-entropy between the output maps and the ground truth. The training algorithm has been configured to stop running either after 200 epochs or after 30 epochs without an improvement on the validation set, whichever condition occurred first. It is important to clarify that our model is trained from scratch for each experimental configuration, without any pre-training on external datasets and without a separate fine-tuning stage. All weights are randomly initialized, which is suitable for ReLU-activated networks. The batch size is 4 images per batch, due to the 512 × 512 input size and GPU memory constraints.

Each dataset used in our experiments has been split into three disjoint sets of images (training set, validation set, and test set) as described in Table 3. For the match experiment, the training set has been used to extract examples for the network, while the stopping criterion periodically measured the performance on the validation set to decide when to interrupt the training procedure. At the end of the training phase, examples from the test set have been used to evaluate the performance of the network. For the mismatch experiment, we used the training and validation sets from the synthetic dataset to build a single network during the training phase. We then evaluated the performance of this single network on the test sets of every other dataset.

4.7 Comparison with state of the art

In order to show that our approach is effective in producing fused localization maps, we compared our approach to a state-of-the-art localization fusion method (DST). Moreover, we also aimed at showing that our fusion technique is not subject to drastic performance drops when tested on different datasets from the one used for training, as opposed to end-to-end deep learning methods. To do so, we performed a comparison with a state-of-the-art deep forgery localization technique (DenseFCN).

DST (Ferrara et al., 2015) is a decision fusion framework based on Dempster-Shafer Theory of Evidence (DST). This approach can be used to combine evidence coming from two or more image tampering localization algorithms. Firstly, the maps produced by each tool are converted to Basic Belief Assignments (BBA) about the presence of the forensic trace the tool looks for. Such conversion step inherently embeds information about the tool's reliability, taking into account both local elements such as brightness and texture, and global elements such as JPEG compression quality. Then, the BBAs produced by multiple tools are fused by using Dempster's combination rule, thus obtaining an aggregated belief assignment. During the fusion step, the framework needs to take into account potential mutual incompatibilities among forensic traces, which imposes strict requirements on the fused localization maps. However, for the forgery localization maps considered in our work, no such incompatibilities are observed. Therefore, the framework we propose offers greater generality and flexibility to handle a wider range of scenarios.

DenseFCN (Zhuang et al., 2021) is an encoder-decoder architecture for image tampering localization in practical forensic applications. The encoder network contains five dense blocks and three 2x2 average pooling layers, and the decoder network contains two dense blocks, three transposed convolutional layers, and a 5x5 convolutional layer. Dense connections and dilated convolutions are used to improve the tampering localization performance. The authors also introduced a strategy to generate large-scale tampered images for training. This strategy uses Photoshop scripting to imitate a realistic tampering process. In this experimental section we used a DenseFCN model⁷ published by the authors, which was trained with Dresden (Gloe and Böhme, 2010) script dataset and fine-tuned with 56 NC2016⁸ images.

4.8 Results and discussion

We report the results for both the match and the mismatch scenarios in Table 4.

Table 4. Performance metrics obtained on the test sets by the single tools, the fusion method proposed by (Ferrara et al. 2015), the method proposed by (Zhuang et al. 2021), and our method.

We report the mismatch case for our method, DST and DenseFCN; we also report the match case for our tool. Values denoted by a * refer to dummy feature maps as images contained in the dataset were not encoded as JPEG.

We can notice how the areas under the ROC curves (AUROCs) obtained by our method are always higher than the one obtained by the best single tool. This suggests that, in this case, our approach is capable of discovering which tool is the most reliable for each example, and at the same time include some clues from the less-reliable ones.

Moreover, the performance in the mismatch case is comparable with the one obtained on the match one. This kind of behavior cannot be taken for granted when dealing with deep learning methods. Neural network training procedures, in fact, are based on the assumption that training and test data follow the same distribution. In real-world applications, however, this is seldom true, as dataset curators can usually perform only a small set of tampering operations on their data, thus unintentionally introducing a bias. On the other hand, a generic fusion method will encounter localization maps extracted from a large number of differently tampered images in the wild. Experimental results confirm that our method has good generalization capabilities that allow it to be used for real-world applications.

It should also be noted that our tool never receives the original image as input. Thus it can only decide the reliability of a tool by examining the spatial distribution of the predicted tampered regions.

In Table 4 and Figure 5 we report the results of a comparison between our method and two state-of-the-art approaches: DST (Ferrara et al., 2015) and DenseFCN (Zhuang et al., 2021). We argue that the real capabilities of each approach are better represented in experiments where the model is tested on a different dataset from the one used for training (mismatch scenario). For this reason, both our method and DST have only been trained on the synthetic dataset. As we did not manage to retrain DenseFCN, we used the pre-trained model provided by the authors instead. The results indicate how testing on a different dataset does not severely affect the performance of the proposed method, as opposed to state-of-the-art approaches.

Furthermore, in Figure 6 we compare, for a set of significant examples, the maps generated by model-based tools, DST, DenseFCN, and the proposed technique. While in many cases DST and DenseFCN struggle to identify the tampered region correctly, our method produces a localization map mostly coherent with the ground truth. We believe that the fusion network is capable of assessing the reliability of each input tool by ignoring areas with incoherent patterns of detection. When producing fused maps for different examples, our tool will weigh the importance of each tool dynamically, thus highlighting their contributions only when they are deemed reliable.

We also compare, in Figure 7, maps produced by our tool in the match scenario with those produced in the mismatch scenario. Training on a different datasets does not seem to significantly reduce fusion performance, and in some cases it may also improve the results. This is due to the fact that the model is less prone to overfitting as it was trained on a much larger dataset (Synthetic).

Moreover, to provide a fair comparison among the evaluated methods, we also analyze their architectural characteristics, number of trainable parameters, and computational requirements. Our proposed method contains approximately 1.9 million trainable parameters, significantly fewer than DenseFCN (8.2 million parameters). This efficiency, combined with better generalization, makes our approach more suitable for real-world forensic applications. Our method requires 18.3 GFLOPs per image, approximately 43% of the computation required by DenseFCN (42.7 GFLOPs).

5 Conclusions

In this paper, we introduced a novel decision fusion method for forgery localization maps. To do so, we exploit a fully-convolutional neural network model capable of aggregating clues from multiple model-based forgery localization tools.

Experimental results show how the proposed method is capable of significantly outperforming the single model-based tools. In order to better simulate a real-world scenario, we also evaluated the performance of our method in a mismatch case where completely different datasets are used during the training and the test phases. In this second scenario, our approach still outperforms all the model-based tools, albeit with lower overall performance. Finally, we compared the proposed method with other state-of-the-art algorithms. Our tool consistently improved over the state-of-the-art in all the datasets.

While the proposed framework demonstrates strong performance and generalization capabilities, several limitations and opportunities for improvement remain. We outline below the most promising directions for future research. A limitation of our current approach is given by the black-box computational model of neural networks. Even though empirical evidence shows that our approach is successful in merging multiple localization maps, in fact, we do not currently have a way of highlighting the areas of input localization maps that have been deemed reliable for taking the final decision. A forensic practitioner would undoubtedly appreciate such a facility, as it would help them assess the confidence with which the neural network took its decision. Future works will aim to improve this aspect by designing a way of highlighting areas where the network turned its attention.

In addtion, our network expects a fixed set of N input maps. However, in practice, not all forensic tools may be applicable to a given image. An improvement would be to design a dynamic fusion mechanism that can handle a variable number of input maps and automatically exclude tools that are unreliable or inapplicable for a particular image. Beyond that, the proposed method could also be extended to allow fusing the output of clone detection algorithms, whose localization maps are typically different in nature as they highlight both the original and the cloned regions.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Marco Arnesano, University of eCampus, Italy

Reviewed by: Gaurav Meena, Central University of Rajasthan, India

Subir Biswas, University of Memphis, United States

Footnotes

IFS-TC Image forensics challenge web page is accessible at https://signalprocessingsociety.org/newsletter/2013/06/ifs-tc-image-forensics-challenge.

Korus realistic tampering dataset is accessible at https://pkorus.pl/downloads/dataset-realistic-tampering.

UMD Face Swap Dataset is accessible at https://mfc.nist.gov/#pills-resources.

SwapMe is no longer available in the iOS App-Store.

FaceSwap repository is accessible at https://github.com/MarekKowalski/FaceSwap/.

IMD2020 Real-Life Manipulated Images is accessible at http://staff.utia.cas.cz/novozada/db/.

DenseFCN model is accessible at https://github.com/ZhuangPeiyu/Dense-FCN-for-tampering-localization.

NIST Open Media Forensics Challenge datasets are accessible at https://www.nist.gov/itl/iad/mig/open-media-forensics-challenge.