The hidden value of uncertainty in medical segmentation

1 Introduction

The challenge of automated analysis in medical imaging has remained a focus for computer vision research for decades, covering a range of tasks and numerous imaging modalities. There is also a strong motivation for practical solutions, as the volume of medical scans produced each year continues to push the capacity of radiologists (Jing et al., 2025). As automated tools near clinical readiness, increased emphasis has been placed on the uncertainty of clinical reference standards (Tejani et al., 2024; Lekadir et al., 2021). In this article, we argue that current representations of reference annotations limit the characterisation of annotation uncertainty and obscure its resulting impact on the reliability of model evaluations.

Though this line of questioning applies to many areas of computer vision, in this article we focus on the task of segmentation: identifying which pixels in an image belong to a given class. From a clinical perspective this involves delineating the boundary of a target, such as an organ or a tumor, with each class boundary commonly stored as a dense binary mask (Sharma and Aggarwal, 2010). Though a binary mask is an efficient representation for computation, many of the structures targeted in the medical domain do not have clear, well-defined boundaries (Pham et al., 2000).

Instead, the binary mask is an expert interpretation, informed by the visual evidence, the annotation protocol, and the prior experience of the annotator. A clinician may not be completely sure whether some pixels belong to the target, in particular those on the boundary, but the limitations of the representation result in this uncertainty being discarded at the point of segmentation. This process of reducing a complex spatial observation into a simplified representation occurs repeatedly as shown in Figure 1, starting when the annotations are first created, and continuing to when the segmentation model is evaluated.

This information can be partially recovered through intra- and inter-annotator variability studies, but when reported this is still commonly summarized as a single scalar measure of annotation similarity (Joskowicz et al., 2019; Yang et al., 2023). Though a single value is easy to compute and use when interpreting model evaluations, this continued reduction represents an irreversible loss of complex spatial relationships. Both representing the reference annotation as a single binary mask and summarizing annotator agreement as scalar values result in the loss of the location, extent, and distribution of annotator variability in benchmarks. This has particular consequences for model evaluation: when multiple expert annotations are plausible, measured performance becomes conditional on the particular reference annotation used. Representing annotations as distributions of plausible boundaries would help preserve this information, allowing for more reliable model evaluations.

In this Perspective, we consider several related but distinct concepts regarding annotation variability, uncertainty, and agreement. We use plausible annotations to describe the range of reasonable expert interpretations of a segmentation target, and annotation uncertainty to describe the uncertainty represented by the range and distribution of these plausible annotations. Inter-annotator variability describes the observed differences between annotations, with (dis)agreement used when referring to the degree or spatial features of similarity or difference between annotations.

Recent work in medical image segmentation has demonstrated a range of methods for estimating model uncertainty, such as producing spatial probability maps to represent model confidence (Kendall and Gal, 2017; Gawlikowski et al., 2023). Methods to learn distributions of plausible segmentations, such as Probabilistic U-Net, PHiSeg, and more recently CaliDiff, have demonstrated the rich information available from multiple annotations, but the development and evaluation of these methods depend on the availability of multi-annotator data (Kohl et al., 2018; Baumgartner et al., 2019; Wang et al., 2025a). By contrast, many approaches for estimating model uncertainty can be developed with singly annotated datasets, removing the equivalent data requirement for development.

The limited availability of multi-annotator data constrains the development of methods for modeling annotation uncertainty, while the representation of this information has additional consequences for model evaluation. In this Perspective, we argue that the loss of information through current annotation representations introduces a distinct source of uncertainty into reported model performance. We describe this as annotation-induced evaluation uncertainty: variation in measured performance arising from the particular annotation used for evaluation, when multiple distinct expert annotations may be plausible. Our focus is therefore not on introducing a new segmentation method, but on examining how current representations of annotations and their variability can irreversibly limit the characterisation of this uncertainty, motivating richer representations to support more reliable model evaluations.

2 Sources of annotation uncertainty

Annotator disagreement is a recurring factor in medical segmentation, and is conventionally measured by intra- and inter-annotator variability studies. Previous studies in medical imaging have identified a range of contributing factors, such as those relating to the imaging modality, those relating to the annotation protocol, or those relating to properties of the segmentation target (Jungo et al., 2018; Rädsch et al., 2023; Hu et al., 2023). Though some of these contributing factors can be mitigated to an extent, there is often some disagreement about the precise location of the boundary contour (Ye et al., 2023). The extent and spatial distribution of this disagreement are task-specific, and may even vary within datasets when considering individual images (Liu et al., 2022).

When annotating a segmentation, a clinician will combine evidence from the image with prior medical knowledge about the structure, such as shape, topology, and adjacency to other structures (Van der Gijp et al., 2014; Nosrati and Hamarneh, 2016). The certainty of these expectations is task dependent, however. If the segmentation concerns an organ (such as the lungs), it will likely have a constrained location, topology, and shape; if it concerns a pathological target (such as lung nodules), there may be more variety in these properties—even its presence may be disputed (Armato III et al., 2011). Observed inter-annotator variability may also reflect annotator-specific factors, with recent work considering the roles of annotator preferences and expertise (Guo et al., 2022; Liao et al., 2024; Ji et al., 2021).

Not all annotator variability is inherent. Though sources such as low imaging contrast may be unavoidable, others, such as an unclear annotation protocol, can be reduced via clear instructions and visual examples (Rädsch et al., 2023). In practice, the overall annotation variability reflects a mixture of inherent and avoidable sources, and while reasonable efforts can be made to reduce the latter, it is unrealistic to expect perfect alignment between annotators. Identifying regions of uncertainty, and consequently accounting for these during evaluation, therefore depends on how annotation uncertainty is represented and measured within datasets.

3 Current representations of variability

When segmentation models are evaluated against a single annotation per image, annotator variability may be summarized as a single scalar value which can be interpreted as a reference for achievable model performance (Li et al., 2026). Given a dataset with an inter-annotator agreement of 75%, a model achieving an agreement of 90% with the test set should not be seen as the model exceeding the performance of an expert. The test set agreement score is conditional on the particular annotations used for each image, and evaluation against a different set of annotations could yield a different result. Though the measured inter-annotator agreement is indicative of the potential variance in the evaluation score, it cannot provide exact bounds or image-specific information.

When multiple annotations are recorded, they may also be combined into a single consensus segmentation through methods such as label fusion or majority voting, with the STAPLE algorithm arguably the best-known method for this (Warfield et al., 2004). The aim of obtaining a single summary annotation may not reflect the clinical reality, in which multiple annotations may be equally plausible due to both intrinsic ambiguity and differences in annotator style (Jungo et al., 2018; Wu et al., 2024). Condensing multiple annotations into a single consensus annotation irreversibly removes information about the extent and distribution of annotator disagreement from the resulting final annotation.

Multiple annotations can also be used as a source of information for quality control, with significant disagreements between annotators used to highlight potential annotation errors. Annotations from multiple sources have also been used to guide model training using estimates of annotation quality (Wang et al., 2025b). In this journal issue alone there are at least two examples of multiple annotations being used for quality control, with disagreement identifying potentially problematic annotations (Guan et al., 2026) (Tian et al., 2026). Including multiple annotations as part of benchmark datasets allows future researchers to independently confirm the quality and agreement of annotations, rather than relying solely on published values.

Despite the value of images with multiple annotations, information about annotator agreement is frequently unavailable in published benchmarks. A recent review of 241 medical image analysis challenges found that 74% did not report the inter- or intra-annotator variability (Reinke et al., 2025). Both data and annotations in medical imaging are expensive to acquire, so the cost of additional annotations may contribute to the sparsity of annotator variability studies (Tajbakhsh et al., 2020). When complete multi-annotator datasets are not viable, annotator variability is instead represented as a summary statistic. Publicly available multi-annotator datasets therefore provide important resources for developing and validating methods that explicitly account for inter-annotator variability. Recent initiatives such as CURVAS demonstrate how multi-annotator datasets can additionally support uncertainty-aware benchmarking and evaluation (Riera-Marin et al., 2026; Riera-Marín et al., 2025).

4 Summarizing similarity statistics

Similarity measures allow for fast comparisons of segmentation agreement, but at a cost: the aggregation of complex spatial relationships into a single value cannot reflect the full pattern of agreement. These measures are used both during the quantification of annotator variability and the evaluation of models against an expert annotation. In both cases the resulting values are aggregated into summary statistics across a set of images, allowing for broader comparisons but in turn further eroding case-specific detail. Interpreting these statistics requires a careful understanding of what is captured by each measure, and how well-suited they are to the targeted structure.

Frameworks such as Metrics Reloaded are useful in guiding researchers toward choosing a complementary set of measures specific to the segmentation task, and highlighting any potential pitfalls (Maier-Hein et al., 2024). Even though popular overlap-based measures such as Dice or Intersection-over-Union have become standard practice for segmentation, these should be used with caution for small or complex target structures. This is because distance from the boundary is not considered when penalizing incorrectly classified pixels, so slight differences in the boundary can disproportionately affect the overall score (Taha and Hanbury, 2015). The suitability of the measure should therefore not only be considered in relation to the target structure, but also to the expected distribution of plausible segmentations.

Combining overlap-based measures with other measures can provide a more nuanced comparison, such as boundary-based measures (Yeghiazaryan and Voiculescu, 2018). Measures such as these may be better suited for accommodating annotator variability: if the distribution of plausible segmentations is expected to predominantly concern uncertain boundaries, then measures that can distinguish whether differences fall within this region may help identify genuine segmentation errors. Using a combination of complementary measures allows summary statistics to represent multiple distinct properties of agreement, but without information on the extent and distribution of plausible annotations the measures cannot distinguish between annotator variability and meaningful segmentation errors.

When multiple annotations are retained, uncertainty-aware methods of evaluation become possible. Generalized Energy Distance (GED) is an example of such a method, allowing distributions of predicted and reference segmentations to be compared rather than individual pairs of masks (Kohl et al., 2018). The CURVAS challenge further demonstrates the use of multiple complementary methods for uncertainty-aware evaluation in a multi-rater setting (Riera-Marín et al., 2025). When evaluating probabilistic predictions, proper scoring rules such as the Brier score can additionally assess the quality of predicted probabilities, while calibration measures assess the alignment of predicted confidence with observed outcomes (Gneiting and Raftery, 2007; Guo et al., 2017). Though these measures complement spatial similarity measures, they remain dependent on the representation of annotation uncertainty used in evaluation.

5 Evaluating against uncertain benchmarks

The reliability of an evaluation depends on the ability of the test data to represent the underlying distribution. One recognized source of evaluation uncertainty is the sampling of test cases, such as how results vary with different test set samples or the variance in scores across the test cases. However, benchmark uncertainty introduces another source of variance: the expert annotations themselves are samples from a distribution of plausible segmentations. We refer to this as annotation-induced evaluation uncertainty: the variability of the reported model performance due to the distribution of plausible annotations.

In a benchmark with high annotator disagreement, a model could be perfectly recreating the segmentations from one annotator, but evaluated against equally plausible segmentations from a different annotator. An example of this can be seen in Figure 2, comparing a model to the annotations of five experts. If the presence of the target, such as a tumor, is disputed between annotators, the test set may include an annotation by an expert who chose to discard it. If the model's prediction includes the presence of the tumor, then this will be seen as an error, even if the segmentation perfectly replicates that of an expert who chose to include the tumor. This is not just a hypothetical example: many cases similar to this can be seen in existing multi-annotator datasets of pathological targets (Armato III et al., 2011; Commowick et al., 2018). In cases such as these, the agreement score does not reflect the plausibility of the model's prediction, as it has no reference for other plausible annotations.

When test data is represented by a single reference annotation it cannot recreate the distribution of plausible annotations. In an ideal setting, each test case would be able to be evaluated against a case-specific probabilistic representation of plausible annotations, rather than assuming a constant distribution of uncertainty across images (Kohl et al., 2019). Since models are able to represent predicted segmentations as probabilistic distributions, rather than comparing a binary mask output from the model, the comparison could instead be between two distributions: the empirical distribution of plausible annotations and the probabilistic model prediction.

6 Model uncertainty

As AI methods for medical applications have approached clinical readiness, the ability to quantify model uncertainty has become increasingly important for developing trustworthy and explainable models (Faghani et al., 2023). This has led to increased interest in methods to represent the model's confidence in making a prediction, so that low confidence decisions can be treated with caution. Bayesian approaches provided a theoretical foundation for this, viewing the network weights as a probability distribution from which multiple plausible sets of weights could be sampled (MacKay, 1992). Comparing the range of predictions from these weights could then capture the model's epistemic uncertainty – the uncertainty caused by the model's insufficient knowledge of the data (Kendall and Gal, 2017).

A more computationally practical approach was demonstrated by MC Dropout, approximating the Bayesian distribution of segmentations without the same cost (Gal and Ghahramani, 2016). The concept of model calibration extended the field of model confidence, establishing approaches to assess whether model confidence was aligned with the accuracy of predictions (Guo et al., 2017). More recent work has considered how multiple annotations can inform model calibration, such as using annotator variability within spatially varying label smoothing (Islam and Glocker, 2021). Though these approaches begin to connect model and annotation uncertainty, the two remain distinct. A model may be well calibrated with respect to a single annotator while failing to represent the broader range of plausible annotations.

A different approach was taken by the Probabilistic U-Net, which used the multi-annotator dataset LIDC-IDRI in order to produce a probabilistic output (Kohl et al., 2018). Though the distributions created by a Probabilistic U-Net and MC Dropout may appear similar, they should not be considered equivalent. The former aims to model the conditional distribution of plausible segmentations from a dataset with multiple annotations, whereas the latter estimates the model's confidence in a solution.

Though classical similarity metrics still consider annotations as a single dense mask, evaluation methods comparing distributions have been well established. Nearly two decades ago (Crum et al., 2006) demonstrated exactly this, for both Dice and the Hausdorff distance. The limitation is not the lack of theory, but the lack of annotated data. Without sufficient information about the distribution of plausible annotations, methods for modeling annotation uncertainty will struggle to be developed. The availability of segmentation data with multiple annotators per image is limited, and without multiple annotations the underlying distribution of plausible annotations is represented by a single observed reference.

7 Discussion

Currently, a key limitation for the field of annotation uncertainty is a lack of data with multiple annotations. Benchmarks may contain large numbers of images annotated once rather than multiple times, with many publicly available datasets failing to provide any annotator variability study at all (Reinke et al., 2025). Practical approaches to addressing annotation uncertainty should anticipate that the financial cost of expert annotations will remain a constraint (Tajbakhsh et al., 2020). Rather than requiring extensive multi-annotator sampling for each case, one potential direction would be to concentrate annotation effort on a subset of cases and investigate whether these annotations can be used to approximate the distribution of plausible annotations across the wider dataset. Validating such approaches would still depend on multi-annotator data to establish whether the approximated distributions reliably reflect the observed expert variability.

The total number of annotators in a study is an important consideration. Work by Joskowicz et al. (2019) on the inter-annotator variability between eleven clinicians on both tumors and organ contours found that using two to three annotators often only captured a fraction of the variability found in a larger group. Rather than treating each additional annotation solely as another reference segmentation, individual annotations could instead be viewed as samples from an underlying distribution of plausible annotations. Increasing the number of annotations provides additional reference points to characterize the distribution, although the number required to do so reliably will depend on the task. This highlights the need for cost-efficient methods to approximate the distribution of plausible annotations. Research in model uncertainty has previously developed practical methods for approximating computationally expensive probabilistic tasks, as shown by MC Dropout; now annotation uncertainty faces a similar challenge: how to approximate the distribution of plausible segmentations efficiently.

Annotator variability has frequently been treated as something to be mitigated, reduced, or otherwise overcome (Warfield et al., 2004; Zhang et al., 2020; Yang et al., 2023). Yet this variability is rich in information about annotation quality, segmentation difficulty, and context for model evaluations. The current representations of annotation uncertainty are reduced over multiple forms of aggregation: multiple observations become consensus masks, pairwise comparisons become scalar values, and individual scores are summarized into average statistics. No single representation preserves every aspect of annotator variability – as with metrics, each serves a purpose. For the task of providing trustworthy evaluation, however, more detailed information may be needed to identify whether model errors fall within the range of plausible expert annotations.

8 Conclusion

In our view, representations for annotations in medical image segmentation benchmarks have not kept pace with developments in modern segmentation methods. While sophisticated methods exist for representing predictive confidence, a single binary annotation does not reflect the full range of plausible annotations. Though consensus annotations and scalar summaries of annotator variability both offer practical solutions to annotator disagreement, they discard information about the distribution of uncertainty both between and within images.

We argue that a growing limitation lies not in the capabilities of segmentation models, but in the restrictive representations of annotations. Annotation uncertainty is more than just a measure of benchmark quality – through what we describe as annotation-induced evaluation uncertainty, it propagates into model evaluation, with reported performance becoming conditional on the particular reference annotation used. Though model predictions can be represented as probabilistic outputs, as long as they are compared to a single binary mask we cannot properly distinguish between plausible differences and segmentation errors.

Future benchmark datasets should place greater emphasis on characterizing annotation uncertainty by acquiring multiple independent annotations, documenting annotation protocols, and preserving rather than eliminating expert disagreement. The number of annotators should be chosen carefully while considering the nature of the target: having two to three annotators may not capture the full range of annotation uncertainty. This variation in expert annotations should be incorporated into evaluation methodology, enabling probabilistic model predictions to be compared against richer representations of plausible annotations. Encoding model uncertainty allows for a more nuanced comparison between predictions and annotations, but cannot be considered a substitute for measured inter-annotator variability.

Substantive progress in segmentation will ultimately depend not only on building more capable models, but also on building better benchmark representations. If segmentation models are expected to represent uncertainty, trustworthy evaluation must account for uncertainty on both sides of the comparison.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Simone Bonechi, University of Siena, Italy

Reviewed by: Elisa Scalco, National Research Council (CNR), Italy

Ali kadhim Jasim, Imam Ja'afar Al-Sadiq University, Iraq

Rosana El Jurdi, Independent Researcher, Melbourne, VIC, Australia