Single-image performance alone does not establish longitudinal technical performance: a paired-error framework for artificial intelligence-enabled quantitative imaging

1 Introduction

AI-enabled imaging systems are commonly developed and evaluated on individual images using task-specific measures of detection, segmentation, or measurement accuracy. These measures may be appropriate for the task being assessed, but they do not necessarily establish whether a system can measure change accurately across visits. When the intended use is longitudinal measurement, the relevant question is not only how well the system performs at each time point, but how errors behave when two measurements are combined to estimate change (Reinke et al., 2024; Maier-Hein et al., 2024).

This distinction follows directly from the mathematics of paired measurement. If change is estimated as the difference between measurements at two visits, its error depends on the errors at both visits and on their relationship to one another. Errors that are similar across visits may largely cancel, whereas errors that vary between visits may create apparent change, obscure true change, or distort its magnitude. Consequently, cross-sectional agreement, repeatability, reproducibility, and responsiveness describe related but different properties of a measurement system (Bland and Altman, 1986; de Vet et al., 2006; Bartlett and Frost, 2008; Mokkink et al., 2021). A system intended to quantify change therefore requires direct evaluation of change error rather than inference from single-image performance alone.

Serial measurement from photographs provides a useful example. Prior work includes methods for longitudinal lesion registration and paired change detection as well as systems evaluated primarily through individual-image segmentation, area accuracy, repeatability, or phantom measurement (Maguire et al., 2023; Zhang et al., 2021; Poon and Friesen, 2015; Foltynski, 2018; Dastjerdi et al., 2019; Privalov et al., 2021; Medela et al., 2026). For these applications, evaluation should establish error in measured change, the relationship between errors across visits, and pair availability: the proportion of scheduled cases that yield valid measurements at both visits. Accurate measurement in individual photographs does not ensure that differences between visits accurately reflect physical or biological change.

Established quantitative-imaging methodology already provides much of the conceptual foundation for evaluating such claims. The Quantitative Imaging Biomarkers Alliance (QIBA) distinguishes cross-sectional from longitudinal performance and defines measurement claims within specified acquisition, processing, and analysis procedures (Quantitative Imaging Biomarkers Alliance (QIBA), 2017; Obuchowski et al., 2016). The QIBA CT Tumor Volume Change Profile provides a direct example of a longitudinal measurement claim with defined acquisition and analysis requirements and explicit interpretation of measured change (QIBACT Volumetry Biomarker Committee, 2022). More broadly, quantitative imaging biomarker (QIB) methodology addresses bias, repeatability, reproducibility, and technical-performance study design, while imaging-biomarker roadmaps distinguish technical validation from clinical validity and utility (Raunig et al., 2015; Sullivan et al., 2015; O'Connor et al., 2017). The need to evaluate change directly is therefore not new. The challenge is to apply these principles consistently when the measurement procedure includes modern AI components.

AI-enabled measurement can vary at several steps, from image acquisition and calibration to model inference and quality control. Model weights, prompts, preprocessing, segmentation, and postprocessing can affect both measurement error and whether a result is obtained. They can also change how closely errors at one visit resemble those at the other. Work on selective prediction has emphasized that accuracy should be interpreted alongside the proportion of cases yielding an output (Geifman and El-Yaniv, 2017), while AI reporting guidance calls for explicit descriptions of model settings and processing procedures (Tejani et al., 2024). Longitudinal evaluation therefore needs to cover the complete measurement procedure, including acquisition, software, and any manual steps.

We propose a framework for evaluating AI-enabled longitudinal measurement through three questions: how accurately does the complete procedure measure change? How often does it return both required measurements, and where do failures occur? Under which acquisition conditions, processing settings, and software versions were these properties established? These questions address the relationship between errors across visits as well as the possibility that low error among successful pairs may coexist with frequent measurement failure.

We examine these questions in two synthetic lesion-area experiments with exactly known geometry and interval change. The repeated-camera experiment compares single-image and paired-change error across combinations of acquisition and processing settings. The acquisition-resampling experiment then tests how that relationship changes when camera pose, distance, and illumination vary between visits. Together, the experiments assess whether single-image performance predicts change-measurement performance under controlled variation. The exact reference and repeatable acquisitions allow these relationships to be studied without uncertainty in true change. The experiments do not rank individual models or establish validity in physical targets or patients.

2 Conceptual development and prior frameworks

2.1 Literature basis

The framework draws on three complementary areas of measurement research. QIBA and quantitative-imaging metrology specify how to evaluate bias, repeatability, reproducibility, and change within a defined measurement procedure (Quantitative Imaging Biomarkers Alliance (QIBA), 2017; QIBACT Volumetry Biomarker Committee, 2022; Raunig et al., 2015; Sullivan et al., 2015; Obuchowski et al., 2016). Work on agreement, responsiveness, uncertainty, and calibration distinguishes the properties being measured and the evidence needed to interpret them (Bland and Altman, 1986; de Vet et al., 2006; Bartlett and Frost, 2008; Kottner et al., 2011; Mokkink et al., 2021; Joint Committee for Guides in Metrology (JCGM), 2008, 2012). AI and diagnostic-reporting guidance addresses suitable metrics, reproducible methods, and failed or indeterminate results (Reinke et al., 2024; Maier-Hein et al., 2024; Geifman and El-Yaniv, 2017; Bossuyt et al., 2015; Tejani et al., 2024). Table 1 shows how these sources inform the proposed reporting requirements.

Table 1. Relationship of the proposed framework to established measurement and reporting guidance.

The proposed framework extends rather than replaces the listed guidance. “Partly” indicates that the concept is present but is not that framework's main focus for longitudinal evaluation. QIB, quantitative imaging biomarker; QIBA, Quantitative Imaging Biomarkers Alliance; COSMIN, COnsensus-based Standards for the selection of health Measurement INstruments; CLAIM, Checklist for Artificial Intelligence in Medical Imaging; STARD, Standards for Reporting Diagnostic Accuracy Studies.

A focused literature search was conducted to identify prior work relevant to longitudinal image-based measurement and to inform development of the proposed framework. PubMed searches addressed AI-enabled skin or wound monitoring, photographic size or area measurement, and longitudinal pigmented-lesion imaging. Titles and abstracts were screened for technical studies evaluating lesion or wound size, area, image correspondence, or change, and reference lists of relevant studies and methodological papers were reviewed for additional sources. The review also incorporated established literature on measurement properties, quantitative imaging biomarkers, uncertainty, validation, and treatment of failed or indeterminate measurements. This was a focused methodological review, not a systematic review of clinical performance.

2.2 Change error is a paired quantity

Let A_*it* > 0 denote the reference area of target i at visit t, and let Â_*it* > 0 denote its estimated area. Define visit-specific log error as

eit=log(ÂitAit).

For two visits, the reference and estimated log changes are

Δi=log(Ai2Ai1), Δ^i=log(Âi2Âi1).

The error in estimated change is therefore

di=Δ^i-Δi=ei2-ei1.

For finite second moments,

Var(d)=Var(e1)+Var(e2)-2Cov(e1,e2).

If the visit-specific error variances are both σ² and their correlation is ρ, then

Var(d)=2σ2(1-ρ).

This identity shows why single-image evaluation is insufficient for longitudinal measurement. The single-visit (marginal) error distributions at the two visits determine Var(e₁) and Var(e₂), but not Cov(e₁, e₂). They therefore do not determine the spread of errors in measured change. Correlation alone is also insufficient because it does not specify the size of the errors at each visit. Longitudinal performance depends on the joint distribution of errors across visits rather than on either visit considered separately. This result follows directly from paired measurement and does not depend on a particular imaging modality, model architecture, or dataset.

Figure 1 illustrates the consequence. Consider two protocols with identical marginal error distributions at each visit and possible errors of −0.05, +0.05. If errors with the same sign occur in the same target at both visits, the resulting change errors are d = 0, 0. If errors with opposite signs are paired across visits, the resulting change errors are d = +0.10, −0.10. The visit-specific error distributions and variances are identical in both protocols, yet the paired-change variance differs. More generally, different relationships between errors across visits can produce different longitudinal performance while leaving both marginal distributions unchanged. No summary calculated separately at each visit can recover this information.

Because the analysis uses logarithms of area ratios, we state the corresponding ratio-scale quantities explicitly. Signed relative error in the follow-up-to-baseline area ratio is

100[exp(di)-1]%.

For a symmetric description of error magnitude around zero on the log scale, we use

100[exp(|di|)-1]%.

This quantity is not equivalent to conventional absolute percentage error,

100|exp(di)-1|%,

because the two differ for negative log errors. Similarly,

100[exp{SD(d)}-1]%

is the multiplicative change corresponding to one standard deviation on the log scale and should not be interpreted as a percentage standard deviation.

2.3 Longitudinal technical performance is distinct from clinical validity

Longitudinal technical performance concerns how accurately a measurement procedure estimates known change under specified conditions. Evaluation should describe the average error, how much errors vary, how large the most extreme errors are, and how often both measurements are obtained. Clinical responsiveness asks a different question: whether the measured change reflects change in the biological or clinical characteristic of interest. That requires appropriate external evidence and hypotheses about the characteristic being measured (Mokkink et al., 2021). Recovering known changes in synthetic lesion area cannot, by itself, establish clinical responsiveness.

We distinguish four kinds of evidence, drawing on established biomarker-validation frameworks and considering the mathematical basis separately (Raunig et al., 2015; Sullivan et al., 2015; O'Connor et al., 2017):

Mathematical basis. Identities and their logical consequences, independent of empirical data.

Controlled computational evaluation. Measurement performance in generated images for which the reference geometry and change are known exactly.

Physical and clinical validation. Evaluation with physical targets whose geometry can be independently verified, followed by human images assessed against an appropriate clinical reference.

Clinical utility. Evidence that use of the measurement improves clinical decisions, outcomes, workflow, or value.

These levels distinguish the claims supported by different kinds of evidence; they are not a mandatory sequence of development. Technical and biological or clinical validation may proceed in parallel in imaging-biomarker research (O'Connor et al., 2017). A computational experiment can establish performance for a known synthetic target and demonstrate a measurement failure mode. It cannot establish accuracy with real cameras, the clinical meaning of lesion boundaries, a clinically important change threshold, or patient benefit. Simulation provides an exact reference for testing measurement behavior, but omits many sources of variation in actual imaging and clinical practice.

2.4 Pair availability is a separate performance property

Longitudinal measurement also requires a valid result at both visits. Let

Rit={1,if the protocol returns a valid measurement for target i at visit t,0,otherwise.

Paired-error summaries describe the distribution of d_*i* conditional on valid measurements being returned at both visits, that is, conditional on R_*i*1 = R_*i*2 = 1. Pair availability is

P(Ri1=1,Ri2=1)

over the prespecified target set.

Error among completed pairs and pair availability answer different questions. A protocol may return accurate measurements for easy cases but fail on difficult ones. Another may return more complete pairs with greater measurement error. These properties should be reported separately unless the intended use provides an explicit way to weigh measurement errors against failures.

Failure counts should identify where processing stopped and avoid counting the same failure more than once. For example, if several measurement methods reuse one marker detection, a failed detection prevents all of them from producing a result. Those missing results share one cause. Reporting that cause and the affected cases alongside successful measurements follows established diagnostic-reporting principles (Bossuyt et al., 2015).

2.5 Longitudinal claims apply to a defined protocol

QIBA Profiles describe performance for a specified measurement procedure, including the equipment, software, and people involved (Obuchowski et al., 2016). Longitudinal AI measurement should be evaluated on the same basis.

The protocol must specify the acquisition procedure and the computational settings that can affect measurement. These include the model architecture and checkpoint, prompts and inference settings, preprocessing, calibration, boundary estimation, postprocessing, quality-control criteria, retry and abstention rules, manual review, software dependencies, and execution environment.

Changes to these steps can alter single-image error, the relationship between errors across visits, or the chance of obtaining both measurements. A performance claim should therefore identify the tested protocol version and the settings in which it is intended to be used. After a substantial change, repeat testing or a comparison across versions is needed to determine whether change measurements remain comparable.

3 Framework for evaluating longitudinal technical performance

3.1 Central hypothesis and testable implications

The variance identity in Section 2.2 establishes that single-image error distributions do not determine error in measured change; no experiment is needed to prove that identity. The experiments instead ask whether this distinction matters when comparing measurement settings: can settings with similar single-image error differ in change error, can their rankings reverse, and can apparently accurate results conceal frequent measurement failure? These questions motivate the first three implications below.

The fourth implication concerns software updates and remains a proposal for future evaluation.

P1. Similar single-image error can accompany different change error. Protocols with comparable errors on individual images may differ substantially when measuring change because errors at the two visits are related differently.

P2. Rankings based on single-image error may reverse when measuring change. A protocol with lower error on individual images may have greater error in measured change, particularly when single-image results are similar.

P3. Accuracy among successful measurements does not establish how often measurement succeeds. A protocol may measure change accurately in the cases it completes while failing to obtain both measurements for many scheduled cases.

P4. Longitudinal performance may change after software updates. Changes to model weights, inference settings, preprocessing, calibration, or postprocessing may alter paired performance even when single-image performance is preserved or improved. Measurements obtained before and after an update should not be assumed interchangeable without a separate comparison across software versions.

3.2 Longitudinal performance measures

The primary quantity of interest is the distribution of paired-change error, d. Its variance is

Var(d),

which decomposes exactly as

Var(d)=Var(e1)+Var(e2)-2Cov(e1,e2).

We also report SD(d) because it is easier to interpret on the log scale. Mean paired log error characterizes systematic bias and can be back-transformed to the ratio scale. Median, interquartile range, and empirical upper quantiles may be reported as additional descriptive summaries when supported by the sample size.

Pair availability is reported separately from error. It is defined as the proportion of scheduled longitudinal cases for which valid measurements are obtained at both visits. Availability should be calculated using the original scheduled cases as the denominator, with the stage of failure reported separately.

3.3 Minimum reporting requirements

A claim about longitudinal measurement should report three things: error in measured change; how often both measurements were obtained and where failures occurred; and the acquisition conditions, processing settings, and intended use. Table 2 lists the information needed to interpret such a claim.

Table 2. Minimum reporting requirements for an AI-enabled longitudinal measurement claim.

The profile complements established quantitative-imaging methodology, AI-reporting guidance, application-specific standards, and applicable regulatory requirements by specifying the evidence needed to evaluate change across visits.

Section 4 tests whether protocols with similar single-image error differ in paired-change error (P1), whether their performance rankings reverse (P2), and whether low error among successful pairs coexists with poor pair availability (P3). These comparisons assess the practical consequences of the variance identity within a controlled imaging system. Comparability across software updates (P4) requires a separate study across versions, as discussed in Section 5.4.

4 Controlled computational evaluation

4.1 Experimental design

The two experiments address complementary questions. The repeated-camera experiment tests whether performance on individual images predicts performance when measuring change across a broad set of processing and acquisition settings. The acquisition-resampling experiment tests whether the resulting error cancellation persists when the camera and lighting vary between visits. Both use synthetic lesions with exactly known geometry and interval change.

4.1.1 Shared targets and measurement procedure

We generated lesions with known area on flat and curved surfaces so that estimated area could be compared with an exact reference. Each lesion and its supporting surface were retained across images, allowing measurements of the same target to be compared between visits. The reference area was calculated from the triangles forming the three-dimensional lesion surface. Section S2 describes the shapes, sizes, contrast settings, and growth procedure.

Measuring lesion area required two inputs: the three-dimensional surface visible in the image and the pixels belonging to the lesion. For the surface, we used either the known geometry from the renderer or geometry estimated by MoGe-2 (Wang et al., 2025) or Depth Pro (Bochkovskii et al., 2025). Each geometry input assigns a three-dimensional position to each pixel. The known renderer geometry is labeled authored geometry in the tables and figures.

For the lesion boundary, we used either the renderer's known lesion mask or a mask estimated by SAM ViT-B (Kirillov et al., 2023) or MedSAM (Ma et al., 2024). A mask marks each pixel as inside or outside the lesion; the renderer's mask is labeled authored mask. Crossing the three surface sources with the three masks allowed us to examine how each contributed to area error.

Eight coded markers of known dimensions provided a length reference for converting the estimated surface to millimeters. We tested six subsets of these markers, varying their number and position around the lesion. These calibration sets are called marker supports in the tables. A scale factor fitted to detected marker corners was applied to both MoGe-2 and Depth Pro. The known renderer geometry needed no rescaling, but all methods still required successful detection of the selected markers. Area was calculated by summing the areas of surface triangles inside the lesion mask.

SAM and MedSAM were given a box around the lesion, calculated from the renderer's mask. We therefore evaluated how well they outlined an already located lesion, not whether they could detect a lesion without assistance. Figure 2 shows the image, calibration markers, box, and known and estimated boundaries. Sections S1–S6 provide the rendering, calibration, and processing details.

4.1.2 Repeated-camera experiment

Seventy-two synthetic lesions were imaged at two visits: 36 had unchanged area, and a separate 36 had 10% area growth. We tested camera-to-surface distances of 15 and 25 cm and image widths of 512, 768, 1,024, 1,536, and 2,048 pixels to vary the amount of image detail available for measurement. Both visits repeated the camera pose, distance, and lighting, with independently generated sensor noise. In the fixed-marker condition, marker positions were also repeated. In the recreated-marker condition, marker centers were shifted independently at each visit (standard deviation, 3 mm). Lesion area changed only in the growth arm. Neither condition reproduced the wider variation in imaging conditions expected during clinical follow-up.

Each configuration specified the marker-placement condition, camera distance, image width, surface source, lesion mask, and calibration marker set. All combinations of these choices yielded 1,080 configurations per arm. The primary analysis used the 960 configurations containing at least one trained AI model. The other 120 combined known renderer geometry with the known lesion mask and served as controls. They were also included in a separate analysis of all configurations. Eligibility was based on the methods used, not on their observed performance. Within each arm, configurations reused the same lesions and images.

Repeating the camera and lighting favors similar errors at the two visits, especially for unchanged lesions processed by deterministic models. In the no-change arm, the 120 controls using known geometry and masks had numerically zero variance in change error: the same approximation of the surface was used at both visits, so its error canceled on subtraction. These controls contain no trained AI model and are excluded from the primary analysis. They are also omitted from ratios that divide by paired-change SD, because that denominator is zero.

4.1.3 Acquisition-resampling experiment

The acquisition-resampling experiment tested whether error cancellation persisted when camera and lighting conditions varied between visits. Each of the same 36 unchanged lesion identities had a baseline image, a matched follow-up with the same camera and lighting but independent sensor noise, and a follow-up with independently sampled acquisition. Baseline and resampled follow-up parameters were drawn independently for each lesion; matched follow-up repeated baseline camera and lighting. Camera distance was uniform from 15 to 25 cm; two coordinates defining viewing direction were independently uniform from −8 to +8 degrees. The camera aimed at the surface origin using Blender's track-axis convention, which also changed image-plane orientation with viewpoint azimuth. Each of three light energies received an independent multiplier with a logarithm uniform from−0.2 to +0.2. Geometry, lesion appearance, and physical marker positions remained unchanged.

The 108 images were processed at a width of 1,024 pixels with the same model checkpoints and six calibration marker sets. This yielded 54 configurations per condition, including 48 with at least one trained AI model. Both acquisition conditions were processed in the same software environment. We compared them using lesions measured successfully in all three images, so that each condition used the same lesions and baseline measurements. We also checked results using every available baseline–follow-up pair within each condition. Uncertainty was assessed with 2,000 bootstrap samples of lesions, using the same samples for both conditions.

The ranges of camera and lighting variation were chosen for the experiment; they were not estimated from clinical photographs. Because these factors varied together, the experiment measures their combined effect. Section S15 provides the full acquisition and processing details.

4.1.4 Error measures and statistical analysis

Error for an individual measurement is the logarithm of estimated area divided by reference area. Paired-change error is the follow-up log error minus the baseline log error. We summarize these errors by their mean, standard deviation (SD), and upper quantiles. The mean describes systematic over- or underestimation, the SD describes the spread of errors, and upper quantiles describe the largest errors. Pooled single-image error SD combines errors from both visits, using only lesions measured successfully at both. Paired-change error SD uses those same lesions. Ratios with paired-change SD in the denominator are calculated only when it is non-zero.

We checked whether missing measurements or the choice of error summary affected the findings. One analysis retained only configurations that measured every scheduled lesion at both visits. Another included every available single-image measurement, even when the other visit was missing. For direct comparisons between configurations, we also recalculated both sets of results using only lesions measured at both visits by both configurations. Finally, we assessed rankings using other measures of error spread, including within-visit SD, median absolute deviation, interquartile range, and an upper quantile of absolute error (Section S8).

The comparisons reuse lesions, images, and processing results, so configurations and their comparisons are not independent experimental observations. To assess how much the findings depended on the particular lesions, we drew 2,000 bootstrap samples of 36 lesions with replacement within each arm. Each sample was applied to every configuration and both visits, with missing measurements kept missing. These exploratory analyses show how the findings vary when the observed lesions are resampled; they do not provide confidence intervals for clinical performance.

4.2 Single-image error does not determine paired-change error (P1)

A common way to estimate the spread of change errors from single-image performance is to assume that errors at the two visits are independent. Under this assumption,

Varind(d)=Var(e1)+Var(e2).

In the no-change arm, assuming independent errors overestimated paired-change SD in all 960 primary configurations, by a median factor of 6.4 (interquartile range, 3.9–9.7). This was expected: the repeated scenes favored similar errors at both visits, which canceled when measurements were subtracted. Only sensor noise and, in the recreated-marker condition, marker positions varied. The experiment demonstrates that single-image and paired-change error can differ substantially (Figure 3A), but does not predict the size or direction of this difference in clinical imaging. The ratio also varied widely across configurations, so no single factor could convert pooled single-image error SD to paired-change error SD.

A worked example uses fixed marker placement, 25-cm working distance, 1,024-pixel images, four surrounding markers, MoGe-2 geometry, and SAM ViT-B boundaries (Figure 4). Its visit-specific log-error variances were 0.0108398 and 0.0094870, with covariance 0.0100126, giving

Var(d)=0.0108398+0.0094870-2(0.0100126)=0.000301671.

The corresponding cross-visit correlation was 0.98735. Ignoring covariance would therefore have produced a paired-error standard deviation 8.2 times larger than the observed value.

We next asked how much change error could differ between configurations with similar single-image error. We compared configurations that differed in one factor and had pooled single-image error SDs within 10% of each other. For marker placement, the median ratio of larger to smaller paired-change SD was 2.11 (Figure 3C). This ratio describes the size of the difference, without indicating which setting performed better. Eligible comparison counts ranged from 22 for camera distance to 1,137 for image width. These counts describe comparisons that reuse the same lesions, not independent observations. Results from the smaller groups are more sensitive to individual comparisons. Some comparisons also included different numbers of successfully measured lesions (Table S7).

The worked configuration showed the same pattern. Changing from fixed to recreated marker placement increased the sum of the visit-specific error variances by only 3.1%, from 0.0203268 to 0.0209492, while paired-change variance increased by 96.6%, from 0.000301671 to 0.000592979. Because marker placement and simulated capture noise changed together, this comparison cannot determine which component caused the difference. It does, however, show that longitudinal error can change substantially even when the spread of single-image errors changes little.

The 10% growth arm showed the same general dependence of paired performance on the relationship between errors across visits, although the discrepancy between single-image and paired-change error was smaller. Detailed results for both change arms are reported in Tables 3, 4.

Table 3. Comparison of single-image and paired-change error across acquisition and processing configurations.

Both arms use the 960 configurations containing at least one trained AI model. Both SDs use lesions measured at both visits within each configuration. The no-change and growth ranking analyses contain 459,990 and 460,018 unordered comparisons with unequal single-image SDs, respectively. Table S5 (continued) also reports results including the controls that use known geometry and masks. Marker-derived scaling applies to both AI geometry models; known renderer geometry retains its original scale but still requires successful marker detection. Configurations reuse lesions and processing results, so their comparisons are not independent. Ratios are omitted when paired-change SD in the denominator is zero.

Table 4. Paired results for the worked configuration.

The example uses a 25-cm camera distance, 1,024-pixel images, four surrounding calibration markers, MoGe-2 geometry, and SAM ViT-B boundaries. The growth and no-change arms used different lesions and were not compared directly. Recreating marker positions also changed capture noise, so this comparison does not isolate the effect of marker placement. Variances and covariances are in squared log units. Table S8 provides further summaries of error for this example.

4.3 Single-image rankings are not always preserved longitudinally (P2)

Across the 960 primary no-change configurations, pooled single-image error SD and paired-change error SD had a Spearman correlation of 0.724. The configuration with lower pooled single-image error SD had higher paired-change error SD in 106,581 of 459,990 comparisons with unequal single-image SDs (23.2%). Including the 120 renderer-only controls raised the correlation to 0.802 and reduced the reversal frequency to 18.8%, illustrating why the analysis population must be stated explicitly.

A lower single-image SD did not guarantee lower change-error SD even when the single-image difference was substantial. When the larger single-image SD was 1.5–2 times the smaller, the configuration with the lower value had higher change-error SD in 23.8% of comparisons. The corresponding frequency was 13.8% for ratios of 2–3 and 4.0% for ratios of at least three (Figure 3B and Table S5). When single-image SDs differed by less than 10%, rankings reversed in 46.3% of comparisons. Each SD was estimated from at most 36 lesions, so such small differences may reflect uncertainty in the estimate.

Nearly tied rankings were unstable when the lesions were resampled. Only 61 of the 22,432 original near-tie reversals retained the same ranking by both error measures in at least 95% of bootstrap samples. A reversal frequency near 50% therefore provides little evidence about which of two nearly tied configurations will measure change more accurately. Comparisons with larger single-image differences were more informative, although their reversal frequencies also varied with the sampled lesions. Resampling assesses this instability; separating the fraction caused solely by estimation noise would require a model of the underlying error distributions.

Using only lesions measured at both visits by both configurations gave an overall no-change reversal frequency of 23.6%. We also checked results after requiring measurements for every scheduled lesion, including unpaired visits in the single-image summaries, using other measures of error spread, and excluding MedSAM (Section S8). In the separate 10% growth arm, the 960 primary configurations had a correlation between the two SD rankings of 0.761, an overall reversal frequency of 22.5%, and a near-tie frequency of 47.6%. These results show the pattern under another change condition within the same synthetic system; they are not an independent replication.

4.4 Pair availability must be reported separately from error among successful pairs (P3)

Across the repeated-camera schedule, 8,231 of 8,640 unique scheduled lesion pairs yielded both measurements after the recorded retry policy, giving final pair availability of 95.3%. One six-scene work unit was externally interrupted during geometry processing and rerun once with identical inputs and environment. The interrupted attempt did not produce first-pass measurement outcomes, so first-pass availability cannot be reconstructed. Final availability was 99.4% under fixed placement and 91.2% under recreated placement. These figures summarize the designed simulation, not clinical failure rates.

All 423 failures occurred during marker detection; no AI component contributed a final failure. Each detection result was reused by nine combinations of surface and boundary methods, producing 3,807 missing measurement rows. Those rows reflect the same 423 failures. Pair availability therefore depended only on marker placement, camera distance, image width, and the selected marker set. Table S12 reports these settings together, counting each marker-detection result once. The results do not establish failure rates for the trained AI models.

Among the 960 primary no-change configurations, 86 combined below-median paired-change error SD with pair availability below 95%. The rank correlation between availability and paired-change error SD was −0.273. These summaries relate error among successful pairs to availability determined by shared marker detections; they should not be interpreted as evidence that individual AI models differed in their probability of failure.

In the no-change arm, the lowest availability occurred with recreated markers, 25-cm distance, 512-pixel images, and four clustered calibration markers: both visits were measured for only 22 of 36 lesions (61.1%). All nine combinations of surface and boundary methods shared those marker detections and therefore had the same missing measurements. This was one pattern of failure shared by nine configurations. Across the no-change experiment, there were nine distinct availability values. Some configurations still had relatively low error among the lesions they measured, but error could not be assessed in the cases they failed to measure.

4.5 Independently resampled acquisition

Varying camera and lighting conditions increased paired-change error SD in all 48 configurations using a trained AI model (Figure 5). Each comparison used the same lesions measured successfully in all three images. We divided resampled-acquisition SD by matched-acquisition SD for each configuration; the median ratio was 6.24 (interquartile range, 2.94–7.04; central 95% bootstrap interval, 4.44–8.94). Across configurations, median paired-change SD increased from 0.0264 to 0.1464 log units. Median pooled single-image SD was 0.1643 under matched acquisition and 0.1484 under resampling. Calculating the difference in single-image SD within each configuration gave a median of −0.0084 log units (bootstrap interval, −0.0203 to 0.0012).

Median error correlation between visits fell from 0.993 to 0.752. Assuming independent errors overestimated paired-change SD by a median factor of 10.59 under matched acquisition and 1.99 under resampling. Errors therefore remained positively correlated, but varying the acquisition reduced the cancellation produced by repeating camera and lighting conditions.

The matched condition yielded 215 of 216 scheduled lesion pairs across the six calibration marker sets (99.5%); the resampled condition yielded all 216 (100.0%). One marker-detection failure at matched follow-up prevented nine combinations of surface and boundary methods from producing a measurement; no AI component failed. Each configuration had measurements from all three images for 35 or 36 lesions. Including every available pair within each condition gave similar results. These findings apply to unchanged synthetic lesions under the camera and lighting variation tested here (Section S15).

5 Discussion

5.1 Principal findings

The experiments show why longitudinal evaluation needs both measurements from each target. Under repeated-camera conditions, single-image error did not reliably predict paired-change error across configurations. Varying acquisition between visits substantially increased paired-change error without a comparable increase in single-image error. These findings illustrate the practical consequences of the established variance identity in Section 2.2.

The repeated-camera experiment helps explain why errors at the two visits must be evaluated together. With camera and lighting conditions held constant, errors were similar across visits and largely canceled when measurements were subtracted. Ignoring that relationship overestimated the SD of change error by a median factor of 6.4. This factor describes the repeated imaging conditions used here; it cannot be used to predict measurement error during clinical follow-up.

Choosing settings by single-image error alone could also be misleading. In 23.2% of no-change comparisons, the configuration with lower single-image SD had higher change-error SD. This occurred even with clear single-image differences: reversal frequencies were 23.8% for SD ratios of 1.5–2 and 13.8% for ratios of 2–3. By contrast, rankings between configurations with nearly equal single-image SDs changed frequently when the lesions were resampled. Those small differences provide weak evidence for choosing one configuration over another.

Changing the camera and lighting reduced that cancellation. In the acquisition-resampling experiment, change-error SD increased in every configuration using a trained AI model, while single-image SD changed much less. Assuming independent errors overestimated change-error SD by a median factor of 10.6 when imaging conditions were repeated and 2.0 when they varied. This comparison shows why accurate individual measurements cannot establish accurate change measurements: the result also depends on how similarly the method errs at the two visits.

Measurement failure was a separate limitation. One combination of acquisition settings and calibration markers measured both visits for only 22 of 36 lesions. All nine surface–boundary combinations shared those missing measurements because the failures occurred at marker detection. These results cannot characterize AI-component failure, but they show why low error among successful measurements must be interpreted alongside the cases that could not be measured.

These findings support the first three implications: individual-image error does not determine change error, rankings can reverse, and accuracy among completed measurements does not capture failures. The results apply to the synthetic system tested here. They do not establish how frequently these patterns occur clinically. P4, concerning comparability across system updates, was not directly tested.

5.2 Contribution of the proposed framework

The contribution is a practical reporting framework for AI-enabled change measurement. QIBA already makes longitudinal claims conditional on specified acquisition and measurement procedures, as illustrated by its CT Tumor Volume Change Profile (Quantitative Imaging Biomarkers Alliance (QIBA), 2017; QIBACT Volumetry Biomarker Committee, 2022). Established biomarker frameworks also distinguish technical performance from clinical validity and utility (Raunig et al., 2015; Sullivan et al., 2015; O'Connor et al., 2017). We apply these principles to the complete AI-enabled workflow, including errors shared across visits and cases in which the workflow fails to produce both measurements.

Table 2 makes those requirements explicit. Reporting change error together with the frequency and cause of failures allows readers to distinguish accurate measurement of a limited set of cases from reliable measurement across the intended sample. Recording the tested acquisition conditions and software settings also makes clear when further validation is needed after a change to the procedure.

This distinction matters when selecting or updating a measurement protocol. Errors that persist across visits may largely cancel, whereas smaller but less consistent errors may distort measured change. A protocol selected solely for single-image accuracy may therefore be poorly suited to longitudinal use. Reporting pair availability alongside error also prevents successful measurements from obscuring failures elsewhere in the scheduled sample.

5.3 Implications for longitudinal study design

Longitudinal validation should begin by defining the quantity to be measured and the conditions under which the measurement will be used. When both absolute size and interval change are clinically relevant, cross-sectional and longitudinal performance should be evaluated separately because they address different questions. The validation dataset should then reflect the types of change the system is expected to measure, including both no-change and nonzero-change conditions. No-change targets test whether the protocol produces apparent change when none has occurred, whereas non-zero-change targets assess recovery of true change across clinically relevant magnitudes and directions. Whenever feasible, the same targets should be evaluated across multiple change states so that observed differences are less confounded by target identity.

The validation procedure should also reproduce the full measurement workflow rather than isolate the AI model from the conditions in which it operates. Acquisition distance, pose, illumination, calibration-marker placement, prompting, quality control, retries, manual intervention, and other operator-dependent steps may all contribute to longitudinal error. Among comparisons with similar pooled single-image error SDs, changing marker-placement condition produced a larger median paired-change SD ratio than changing geometry or boundary inputs. Marker positions and capture noise changed together, so this comparison does not isolate their individual effects. Although this does not imply that acquisition will dominate in other systems, it illustrates the importance of incorporating the sources of variation expected in actual use.

The reference standard and sample size should suit the claim being evaluated. If the reference is itself measured, its uncertainty must be included. Sample-size requirements differ for estimating average error, variance, and the frequency or size of extreme errors. Claims should reflect how precisely the available longitudinal cases can estimate the chosen quantity.

Missing measurements should remain part of the performance assessment. Error among completed pairs describes accuracy when both measurements succeed; pair availability describes how often they succeed. If failures depend on lesion difficulty or patient characteristics, both should be examined in the relevant subgroups. When the intended workflow includes retries or manual correction, report how often these are needed, how often they succeed, their resource requirements, and the resulting measurement performance.

5.4 Comparability across system updates

Longitudinal measurement introduces an additional challenge when images from different visits are processed under different protocol versions. Changes in model weights, segmentation methods, prompt generation, preprocessing, calibration, software dependencies, or acquisition procedures may alter the relationship between errors across visits even when single-image performance improves.

Neither experiment directly evaluated software-version changes, so P4 remains a proposed implication rather than an empirical finding. The configuration results nevertheless illustrate why similar single-image performance across versions cannot, by itself, establish longitudinal comparability. Studies intended to support measurements across protocol versions should therefore process the same images or targets under both versions and compare the resulting change estimates.

Reprocessing earlier images with an updated protocol may reduce differences caused by using multiple versions, but comparability still needs to be demonstrated. The original images, calibration information, software versions, and processing instructions must be retained so that measurements can be reproduced.

5.5 Path toward physical and clinical validation

Physical validation should test repeated images of flat and curved targets with independently measured geometry. Applying known increases and decreases in area to the same targets would help separate the effect of true change from differences between targets. Imaging should vary across devices, operators, sessions, camera positions, and lighting conditions relevant to the intended use. Reference measurements must include their uncertainty, and lesion-location prompts should come from the intended workflow rather than the known reference boundary.

Clinical validation must also address sources of uncertainty that synthetic and physical targets cannot reproduce. These include matching the same lesion across visits, uncertainty in lesion boundaries on human skin, patient- and operator-dependent acquisition variation, differences in skin and lesion characteristics, missing or failed measurements, generalizability across settings and populations, and the magnitude of error that is acceptable for the intended clinical use.

Technical validity alone would still be insufficient to establish clinical utility. Even an accurate measure of lesion change does not show that using that measurement improves diagnosis, treatment decisions, workflow, patient outcomes, or value. Those questions require separate clinical evaluation.

5.6 Limitations

Both experiments used generated lesions and images with exactly known reference geometry. They omitted features of clinical photography such as patient motion, hair, occlusion, inflammation, lens distortion, and device-specific processing. The virtual camera and noise model were not calibrated to a smartphone. All lesions used 192 boundary vertices and 20 interior rings, limiting the family of shapes represented. The segmentation models were also given boxes derived from the known lesion masks. Their results therefore describe outlining an already located lesion under controlled conditions, not autonomous clinical detection and measurement.

The tested configurations were chosen for controlled comparisons, not sampled from a population of imaging systems. They reuse the same lesions, images, and processing results. The 23.2% no-change reversal frequency therefore describes this experiment: including controls changes it to 18.8%, and comparing configurations on the same successfully measured lesions changes it to 23.6%. Resampling also shows that results depend on the particular lesions included. None of these frequencies estimates how often rankings would reverse clinically. The growth arm uses the same synthetic framework and is not an independent replication.

The fixed and recreated conditions in the repeated-camera experiment repeated camera pose, distance, illumination, and surface appearance across visits; lesion area changed only in the growth arm. Their strong positive error correlation was therefore expected, and neither the size nor the direction of the error from assuming independence can be assumed to apply to clinical photography. Marker positions and noise differed together between the two placement conditions, preventing attribution to marker replacement alone. The acquisition-resampling experiment varies a specified set of imaging factors, but its bounded perturbations are not a measured distribution of clinical acquisition conditions.

Only one non-zero change magnitude was evaluated, and the no-change and growth conditions involved different lesion identities. This limits direct assessment of how measurement performance varies with the magnitude of true change. Future studies should, where feasible, apply multiple positive and negative changes to the same targets.

The computational reference was treated as exact, so the study did not evaluate the effects of reference uncertainty. In addition, paired error could not be observed when either visit failed, preventing assessment of whether failed cases would have had systematically different measurement error.

The results establish that single-image and longitudinal performance can diverge within the evaluated computational system. Their magnitude and clinical significance remain questions for physical and human imaging studies; the present data do not justify universal performance thresholds.

6 Conclusion

AI-enabled systems intended to measure change require direct longitudinal evaluation. Single-image accuracy does not establish how errors combine across visits or how often both measurements will be obtained. The repeated-camera experiment demonstrates differences between single-image and paired-change performance across configurations; the acquisition-resampling experiment shows how strongly error cancellation can depend on camera and lighting conditions.

A longitudinal performance claim should report paired-change error, pair availability and failure stage, and the acquisition and processing conditions evaluated. These synthetic experiments support that approach but do not establish physical or clinical performance. Validation with physical targets and human images, followed by evaluation of clinical utility, is needed before such measurements can support patient-level decisions.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. OpenAI GPT-5.6 was used to assist with language editing, grammar and clarity review, and development of code used to support procedural 3D model generation in Blender. All scientific decisions, analyses, interpretations, and reported results were reviewed and verified by the author(s), who take full responsibility for the content of the article.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fimag.2026.1976334/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Monica Bianchini, University of Siena, Italy

Reviewed by: Ali Golestani, University of Calgary, Canada

Carlotta Fontana, University of Salerno, Italy