A same-condition decision-risk audit for CPU-only machine-learning efficiency measurement

1 Introduction

CPU-only machine-learning efficiency measurement remains important even in an era dominated by accelerators. CPUs still run complete inference tasks in small, educational, and edge settings, and they assist with preprocessing, data movement, and coordination in systems that use both CPUs and accelerators. Efficiency claims in these settings are commonly expressed through wall-clock time, latency, throughput, estimated GFLOP/s, effective bandwidth, or speedup ratios. Large benchmark efforts such as MLPerf Inference show that measured machine-learning performance depends on interactions among workload design, hardware, software stack, numerical representation, power, and deployment scenario (Reddi et al., 2020). Production inference studies reach a similar conclusion: observed efficiency is shaped not only by model architecture, but also by batching, memory behavior, runtime configuration, and platform characteristics (Park et al., 2018). A reported CPU-only efficiency value is not determined by the algorithm or framework alone; it also depends on the measurement condition under which it was obtained.

The problem is that deterministic computation does not imply deterministic timing. Wall-clock time, process CPU time, throughput, estimated GFLOP/s, and effective bandwidth are produced by measurement procedures that can be sensitive to runtime state, operating-system activity, thread configuration, memory behavior, library scheduling, warm-up policy, and execution context. Foundational performance-measurement research has shown that apparently incidental experimental choices can change measured results and bias conclusions (Mytkowicz et al., 2009). Georges et al. (2007) showed that run-to-run performance variation can make single timing numbers misleading, and recommended repeated measurements with confidence intervals. Kalibera and Jones (2013) addressed the practical cost of such repetition by developing methods for choosing how many benchmark iterations, executions, and builds to run within a feasible experiment time. For CPU-only machine-learning efficiency evaluation, the implication is that repeatable computation should not be treated as evidence of repeatable timing. Even when the recorded outputs are consistent, timing-derived quantities such as throughput, estimated GFLOP/s, and effective bandwidth still require repeated-measurement evidence.

For this reason, a claim that one CPU-only measurement condition is faster or more efficient than another should not be interpreted from the reported number alone. The claim also depends on how that number was produced: the execution context, whether early warm-up measurements were included or excluded, whether the value came from one observation or from an average, and how many repeated measurements were available. A result based on all measurements can differ from a result based only on post-warm-up measurements. Similarly, a result based on one timing observation can differ from a result based on repeated observations. The relevant question is therefore not only what difference was reported, but whether a difference of that size is larger than the variation observed when the same documented condition is measured and summarized in the same way.

This paper addresses that question by introducing a same-condition decision-risk audit for CPU-only machine-learning efficiency measurement. In this study, an audit means a structured check of the measurement record rather than a formal pass–fail certification. The decision being supported is the reporting decision: whether a claimed timing or derived-rate difference should be stated strongly, stated conditionally, or worded cautiously. The audit estimates how often repeated measurements of the same documented condition produce a difference above a practical relative-difference threshold under the same phase policy, measurement-block structure, and reporting rule. The study evaluates this audit using eight CPU-only workload scenarios across two local Windows machine contexts and a managed Google Colab CPU-allocation context.

The paper makes three contributions:

• It formulates CPU-only machine-learning efficiency reporting as a measurement-evidence problem. A reported timing or derived-rate difference is treated as a claim that needs support from repeated measurements, not as a standalone property of the algorithm, framework, or workload.
• It defines a same-condition decision-risk audit for judging reported efficiency differences. The audit compares estimates from repeated measurements of the same documented condition and estimates how often the measurement process itself produces differences above practical relative-difference thresholds such as 5%, 10%, or 20%.
• It demonstrates the audit on eight CPU-only workload scenarios and shows that repeatable recorded outputs can coexist with unstable timing-derived efficiency values. The empirical results indicate that differences near common practical relative-difference thresholds need phase-specific, block-aware repeated-measurement evidence and sample-size context for strong support.

The remainder of this paper is organized as follows. Section 2 reviews related work on performance measurement, CPU-side machine-learning efficiency, execution-context variability, warm-up and phase policy, and statistical claim support. Section 3 describes the measurement design, workload suite, execution contexts, measurement conditions, measured quantities, and same-condition A/A audit procedure. Section 4 presents the empirical results and their interpretation. Section 5 discusses implications for CPU-only efficiency reporting and summarizes the main validity limits. Section 6 concludes the paper.

2 Related work

This paper is positioned at the intersection of performance measurement, CPU-side machine-learning efficiency, execution-context variability, phase-sensitive benchmarking, and no-change reference checks. Prior work in these areas shows that timing values are not self-interpreting, that CPU execution remains an important part of machine-learning systems, and that execution context, warm-up policy, repetition structure, and practical relevance affect how efficiency claims should be read. The gap addressed here is more specific: a reported CPU-only timing or derived-rate difference needs an audit that asks whether a difference of that size is larger than the variation produced by the same measurement procedure under the same documented condition.

2.1 Performance measurement and statistical claim support

Performance-measurement research has long shown that seemingly minor experimental details can change measured results. Mytkowicz et al. (2009) demonstrated that factors such as link order, environment size, and setup randomization can introduce measurement bias even when nothing appears obviously wrong. Georges et al. (2007) argued that run-to-run variation in performance measurements should be handled statistically rather than reduced to isolated best, worst, or average values. Kalibera and Jones (2013) extended this concern by showing how repetition can be planned across benchmarking levels, such as iterations, executions, and builds, when experiment time is limited. More broadly, Kounev et al. (2025) present systems benchmarking as an integrated discipline spanning metrics, workloads, measurement methodology, statistical measurement, and experimental design.

This literature supports a central premise of the present paper: a timing value must be interpreted through the measurement procedure that produced it. The issue becomes more important when wall-clock time is converted into throughput, estimated GFLOP/s, effective bandwidth, or speedup ratios, because uncertainty in the timing measurement propagates into the derived efficiency value. Hoefler and Belli (2015) emphasized that performance reporting should include enough information for readers to assess uncertainty and practical meaning. Kalibera and Jones (2020) focused on confidence intervals for effect sizes in performance comparison, while Tullos et al. (2021) distinguished statistical detectability from practical relevance. Chen and Revels (2016) further showed that timer error, operating-system jitter, and environmental noise can lead to non-ideal timing distributions, motivating robust analysis rather than unqualified point estimates.

These studies establish why performance results need statistical and practical context. What remains missing is a simple reporting audit for judging whether a CPU-only efficiency difference is large relative to variation observed under the same measured condition.

2.2 CPU-side machine-learning efficiency measurement

Machine-learning performance is shaped by workload design, deployment scenario, hardware, software stack, numerical representation, and metric choice. MLPerf Inference (Reddi et al., 2020) was developed to make inference benchmarking more representative and comparable across diverse systems, scenarios, and implementations. Its design highlights why machine-learning efficiency cannot be summarized by a single hardware-independent number: inference systems differ in latency constraints, throughput targets, model quality requirements, deployment settings, frameworks, libraries, and platform assumptions. Production-scale inference studies reach a related conclusion. Park et al. (2018) showed that deep-learning inference in data-center settings is shaped by model architecture, batching, memory behavior, runtime configuration, and hardware–software co-design concerns.

CPU execution remains relevant even when accelerators dominate many high-performance machine-learning settings. CPUs still support preprocessing, host coordination, memory movement, runtime orchestration, edge execution, and fallback or hybrid execution paths. Work on LLM inference also shows that CPU-side performance depends on workload structure, model size, memory behavior, quantization, and optimized kernels (Na et al., 2024; Shen et al., 2023). Edge-oriented machine-learning studies similarly emphasize that latency, throughput, power, and available parallelism depend on the device and workload being measured (Jayanth et al., 2024; Khdr et al., 2026).

More directly, CPU-oriented machine-learning studies show that runtime configuration and repeated timing matter. Hasabnis (2018) evaluated TensorFlow CPU backends and showed that threading policy can substantially affect performance. Huber et al. (2025) measured neural-network inference response times on embedded CPU platforms using repeated invocations and both single- and multi-threaded TensorFlow Lite execution. These studies support the need to document CPU-side runtime conditions and to treat timing as a measured quantity rather than as a deterministic property of the model or framework.

The present paper does not propose another benchmark suite or rank CPU platforms. Instead, it uses CPU-only workload scenarios as a measurement surface for studying whether reported timing-derived efficiency differences are supported by repeated-measurement evidence.

2.3 Execution context as part of the measured condition

Execution context can affect measured performance through hardware heterogeneity, virtualization, resource sharing, operating-system behavior, runtime configuration, and provider-specific infrastructure. Gent and Kotthoff (2011) reported reliability concerns when computational experiments are run on virtualized hardware. Leitner and Cito (2016) found systematic variation across public infrastructure-as-a-service providers, instance types, and tenancy modes. Laaber et al. (2019) showed that detecting software-performance changes in cloud microbenchmarking depends on provider, instance type, benchmark, and expected effect size. Methodological guidance on cloud-performance evaluation therefore emphasizes repeated measurement, transparent configuration reporting, and statistical analysis (Papadopoulos et al., 2021).

Recent studies reinforce this point for both cloud and AI-enabled systems. Ahuja et al. (2025) examined performance variability in public-cloud environments using CPU, memory-bandwidth, and I/O-oriented benchmarks. Rahman et al. (2025) showed that environment configurations can affect the stability of AI-enabled systems. These findings support treating configuration and context metadata as part of the measured condition, rather than as optional background information.

Managed notebook environments add a related concern. Carneiro et al. (2018) analyzed Google Colaboratory as a cloud-based environment for machine-learning and deep-learning applications, with attention to available resources, performance, and limitations. That work does not provide a same-condition A/A audit, but it supports the decision to treat Colab as a managed execution context whose allocation and software details must be recorded. In the present study, the local Windows machines and the managed Colab CPU allocation are therefore interpreted as documented execution contexts, not as interchangeable representatives of a single generic CPU platform.

2.4 Phase policy, block structure, and same-condition reference logic

Warm-up and phase behavior are important because different subsets of the same measurement record can support different reported values. Barrett et al. (2017) showed that virtual-machine benchmarks do not always reach a stable peak-performance state, even under controlled conditions, and that assuming convergence can be misleading. Traini et al. (2023) raised similar concerns for Java microbenchmarking, while Beseda et al. (2026) proposed a kernel-based approach for detecting steady state in performance time series. For Python workloads, Crapé and Eeckhout (2020) argued that benchmarking methodology should distinguish start-up behavior from steady-state behavior.

These studies motivate treating phase policy as part of the efficiency claim. A summary based on all accepted measurements and a summary based only on post-warm-up measurements are not merely two presentations of the same evidence; they may describe different parts of the measurement process. This matters for CPU-only machine-learning workloads because early observations, framework initialization, memory behavior, session effects, and runtime scheduling can change the dispersion pattern. The present paper therefore records and reports the phase policy used to form each timing summary.

Repetition structure also matters. Measurements collected within the same session, run, or execution block may be more similar to one another than measurements collected in separate blocks. Winkler et al. (2015) showed that unrestricted permutation is inappropriate when observations are repeated, paired, nested, or otherwise dependent, and proposed exchangeability blocks to preserve known dependence structures. The present paper does not adopt their inferential procedure directly. It uses the same design principle: a no-change comparison should preserve the dependence structure created by the measurement design.

No-change comparisons provide the final methodological precedent for the audit. In online experimentation, A/A validation compares groups intended to be equivalent in order to check assignment, metric behavior, variance estimation, and the analysis pipeline before treatment effects are interpreted (Zhao et al., 2016; Kohavi et al., 2022; Larsen et al., 2024). Empirical-null and negative-control methods use related logic: they estimate a reference distribution from comparisons where the expected effect is absent and use that distribution to calibrate interpretation (Efron, 2004; Schuemie et al., 2014; Yang et al., 2024).

This paper adapts that no-change logic to CPU-only machine-learning efficiency measurement. Repeated measurements of the same documented condition are used as the empirical reference for the timing, phase-selection, block-sampling, and reporting pipeline. The resulting same-condition A/A threshold-exceedance rate does not certify a benchmark as valid or invalid. It gives readers a direct way to judge whether a claimed timing or derived-rate difference is large relative to the variation produced by the measurement procedure itself.

3 Materials and methods

This section describes how the CPU-only measurements were collected, organized, and recorded for the same-condition A/A decision-risk audit. The design covers the workload scenarios, execution contexts, measurement conditions, timing quantities, derived efficiency metrics, and measurement-block structure used in the analysis. The focus is the measurement evidence used to interpret CPU-only timing and derived-rate differences.

3.1 Measurement organization and scope

The main measurement grid was organized hierarchically by execution context, measurement condition, repeated execution, session block, and run configuration. Three execution contexts were studied, and the controlled and reference measurement conditions were evaluated separately within each context. For each context–condition combination, 15 repeated executions were performed, with three session blocks in each repeated execution. Within each session block, the grid contained eight workload scenarios at each of three requested thread counts, and each resulting run configuration had a target of 40 accepted timing observations. Here, an accepted timing observation is a successfully completed workload operation with valid wall-clock and process-CPU timing; deferred, invalid, and failed attempts are retained separately in the attempt-event record (Section 3.4).

The experiment uses three CPU-only execution contexts: two local Windows machines, denoted Machine 1 and Machine 2, and Google Colab CPU allocations. Machine 1 and Machine 2 are treated as separate local physical-machine contexts because they differ in processor, memory, Python version, and numerical-library stack. Colab is treated as a managed cloud-notebook CPU-allocation context because independent sessions may receive different underlying processors; allocation-specific processor metadata were therefore recorded rather than assumed fixed.

The same workload grid is evaluated under two measurement conditions, referred to here as the controlled and reference conditions. The controlled condition includes a pre-execution CPU-utilization gate: before each target timing observation, system CPU utilization is sampled, and the observation is deferred when the sampled value exceeds the configured threshold. Deferred observations are logged as deferred attempts and retried after a bounded backoff. The controlled condition also applies the configured warm-up exclusion, inter-session delay, and post-measurement pacing. The reference condition retains the same workload grid and timing-measurement procedure but disables these selected interventions. The two conditions are matched measurement configurations. Their contrast is descriptive because several design elements differ at the same time; it should not be interpreted as the isolated causal effect of any single intervention.

Fixed seeds, synthetic inputs, CPU-only runtime requests, and deterministic framework controls, when supported by the relevant framework, improve functional-output summary consistency and comparability. These controls are not expected to make timing deterministic. The analysis therefore focuses on the timing and derived-metric variation that remains after these controls are applied.

3.2 Workload suite and measurement roles

The workload suite provides controlled CPU-only measurement targets for four machine-learning-relevant roles: dense numerical computation, memory movement, framework-level inference, and framework-level single-step training. The five workload definitions are NumPy matrix multiplication, NumPy memory streaming, PyTorch multilayer-perceptron inference, TensorFlow multilayer-perceptron inference, and TensorFlow single-step training. These definitions produce eight configured workload scenarios because the NumPy matrix and memory-streaming workloads are evaluated at multiple sizes (Table 1). The workloads are synthetic by design. They are used to study CPU-side measurement behavior under controlled conditions while avoiding additional sources of variation from datasets, data loading, preprocessing, augmentation, file I/O, checkpointing, and full training pipelines.

Table 1. CPU-only workload suite and measurement roles.

The memory-streaming workload uses NumPy array assignment to represent a controlled memory-movement case within the workload suite. It provides a simple CPU-side workload for examining timing and effective-bandwidth variation using the same timing, repetition, and recording procedure applied across the study. The TensorFlow training scenario represents one optimizer-backed update step, which is included as a controlled training-related workload. A PyTorch training counterpart is therefore not included.

3.3 Execution contexts and measurement grid

Table 2 summarizes the hardware and software characteristics recorded for each execution context. These characteristics are part of the measurement record because timing variability can differ across execution contexts. A reported efficiency difference may therefore be easier to interpret in one context than in another. The same-condition audit interprets repeated measurements within each documented context rather than assuming a universal CPU environment.

Table 2. Recorded execution-context characteristics.

The main measurement grid used seed 42 and yielded a target of 43, 200 accepted timing observations for each context–condition combination. Additional controlled-condition diagnostic runs examined sensitivity to the pre-execution CPU-utilization gate. These runs used one representative scenario from each workload definition at requested thread count 1 while varying the gate threshold across no gate, 60%, 80%, and 95%. They were analyzed separately from the main measurement grid to describe how gate settings affected deferment rates and timing variation and were not included in the target accepted-observation count of 43, 200 per context–condition combination.

The experiments were configured for CPU-only execution by disabling visible GPU devices where applicable and applying the intended thread count through common numerical-library environment variables and available PyTorch and TensorFlow thread-control APIs. Runtime and thread-control metadata were retained to document the software configuration. These controls define the intended CPU-only measurement setup, while the recorded metadata document the configuration under which the timing observations were collected.

3.4 Controlled and reference measurement conditions

Each run configuration was executed in a separate measurement process. This structure reduced residual state across workload and thread configurations and allowed runtime settings to be applied before the relevant numerical and machine-learning libraries were configured. For each run configuration, the measurement record retained the configuration, environment metadata, accepted timing observations, attempt-event records, functional-output summaries, and runtime diagnostics. Table 3 lists the design elements that distinguish the controlled and reference conditions.

Table 3. Settings for the controlled and reference measurement conditions.

The CPU-utilization gate is a pre-execution screening step. Before each target timing observation, system CPU utilization was sampled over 0.05 s and memory state was recorded. When the gate was active, an attempt whose sampled CPU utilization exceeded the configured threshold was recorded as deferred, followed by a bounded backoff before retrying the same target timing observation. The backoff was min(5.0, 0.05(1 + k)) seconds, where k is the number of deferrals already recorded for the same target timing observation. The measurement process enforced limits of three invalid executed attempts, 1,000 deferred attempts, 300 s of elapsed retry time per target timing observation, and a six-hour measurement-process timeout.

A timed attempt was accepted only when the operation completed successfully and produced positive wall-clock time and nonnegative process CPU time. Invalid timing deltas and execution exceptions were logged as invalid or failed attempts. Each run configuration continued until the target accepted count was reached or the retry, waiting, or timeout limit was met. The attempt-event denominator was retained as

Nattempt=Naccepted+Ndeferred+Ninvalid+Nfailed.(1)

Metric summaries used accepted timing observations, whereas deferment, invalid-attempt, failed-attempt, completion, and denominator diagnostics used the corresponding attempt-event or target-observation denominators. Timer-resolution diagnostics were recorded for context; across the recorded environments, the estimated minimum nonzero wall-clock timer delta was approximately 100–500 ns on the local Windows runs and 100–308 ns on the Colab runs, several orders of magnitude smaller than the shortest observed primary-grid wall-clock measurement.

Hashes of functional-output summaries were computed for every accepted timing observation from a normalized representation of the workload-output summary. The normalization excluded timing and diagnostic fields, sorted keys, rounded floating-point values to 10 decimal places, and applied SHA-256 hashing. A single unique hash within a completed run configuration indicates repeatability of the recorded output summaries. It does not imply bitwise identity of every intermediate operation and does not imply timing stability. Figure 1 summarizes the measurement lifecycle that produces the measurement record.

3.5 Measured quantities and derived metrics

The directly measured quantities are wall-clock time and process CPU time, both recorded in nanoseconds. The process-CPU/wall-time ratio is

Rcpu/wall=100×Δtcpu,nsΔtwall,ns.(2)

Values above 100% can occur when native libraries or frameworks use multiple CPU threads during an elapsed wall-clock interval; they are process-CPU diagnostics, not errors.

Throughput is reported as samples per second for batch-oriented framework workloads:

Throughput=BΔtwall,ns/109,(3)

where B is batch size. Estimated GFLOP/s is reported for matrix multiplication and framework workloads as

GFLOP/s^=F^Δtwall,ns/109×10-9,(4)

where F^ is the nominal workload-level floating-point operation count. NumPy matrix multiplication uses F^=2n3. Framework inference uses

B×2(784×128+128×64+64×10).

TensorFlow training uses

B×3×2(32×64+64×32+32×1),

a workload-specific approximation for forward, backward, and update-related work. This formula is only an approximate normalizer for reporting TensorFlow training GFLOP/s. The timing results and same-condition A/A analyses are based on the recorded timing measurements, not on the exactness of this operation count.

Effective memory bandwidth for the memory-streaming workload is computed as

BandwidthGB/s=2SΔtwall,ns/109×10-9,(5)

where S is array size in bytes and the factor of 2 represents one read from the source array and one write to the destination array. The estimated GFLOP/s and effective-bandwidth values are workload-specific derived metrics, not hardware peak-performance measurements.

Runtime covariates and process counters are retained as diagnostic metadata. These metadata include pre- and post-execution CPU and memory utilization, resident-set-size changes, and peak memory observations. Additional diagnostic fields record allocation or object-count proxies where available, garbage-collection activity, process I/O deltas, context-switch deltas, session index, run order, requested thread count, and CPU-utilization gate setting. These variables are interpreted as measurement-record metadata and associative diagnostics.

3.6 Same-condition A/A decision-risk audit

This subsection defines how repeated measurements are used to interpret a claimed CPU-only efficiency difference. The audit compares an observed timing or derived-rate difference with variation obtained when the same documented condition is repeatedly measured and summarized under the same phase policy and reporting rule. The audit produces an evidence summary for the stated metric, workload scenario, execution context, practical relative-difference threshold, phase policy, and reporting rule.

3.6.1 Audit unit and required measurement record

A measured condition c is defined by execution context, controlled/reference measurement condition, workload scenario, configured workload parameter, requested thread count, CPU-utilization gate setting where applicable, metric, phase policy, and reporting rule. Accepted timing observations for condition c, metric m, and phase policy p are organized into recorded measurement blocks b. A measurement block refers to the accepted timing observations collected for one run configuration within one session block of a repeated execution. The block structure is retained because measurements collected in different session blocks or repeated executions need not be exchangeable with measurements collected within the same recorded block.

Each measured condition requires the following record:

• Accepted timing observations and derived metrics;
• Attempt-event counts, including accepted, deferred, invalid, and failed attempts;
• Workload role, configured workload parameter, requested thread count, and random seed;
• Execution context and software/runtime metadata;
• Controlled/reference measurement condition and CPU-utilization gate setting where applicable;
• Measurement-block identifiers and measurement order;
• Phase-policy definition;
• Reporting rule used to form the stated efficiency estimate;
• Functional-output summary hashes.

For the block-preserving A/A analysis, the representative subset includes one scenario at requested thread count 1 from each workload role. These scenarios are 100 MB memory streaming, 500 × 500 NumPy matrix multiplication, PyTorch multilayer-perceptron inference with batch size 32, TensorFlow multilayer-perceptron inference with batch size 32, and TensorFlow single-step training with batch size 16. These five scenarios define the representative workload set used in the A/A and sample-size summary tables. The remaining workload-parameter settings and requested thread counts are retained for dispersion, context, phase-policy, and measurement-procedure diagnostic analyses.

This record supports two related interpretations. The first is functional-output summary consistency: whether the workload produced the same recorded output summary under the configured measurement task. The second is timing-evidence strength: whether repeated measurements of the same documented condition can produce a timing or derived-rate difference as large as the claimed one. The output hashes address the first interpretation at the recorded-summary level. The same-condition A/A analysis and sample-size calculations address the second.

3.6.2 Phase policies and reporting rules

A phase policy selects which accepted timing observations are eligible for a timing or derived-rate estimate. The full-run phase uses all accepted timing observations. The configured phase applies the warm-up exclusion defined by the measurement condition: the first five accepted timing observations are excluded for the controlled condition, and no warm-up exclusion is applied for the reference condition. Sensitivity summaries also evaluate exclusions of 0, 5, 10, and 20 accepted timing observations.

A reporting rule defines how the eligible timing observations are summarized before a timing or derived-rate value is reported. Single-observation reporting uses one accepted timing observation. Mean-of-k reporting uses the arithmetic mean of a contiguous ordered window of k accepted timing observations. Median-of-k reporting uses the median of such a window. The audit evaluates the same reporting rule that would be used to make the efficiency claim. This alignment is necessary because single-observation reporting, mean-of-5 reporting, and mean-of-10 reporting can have different same-condition threshold-exceedance behavior.

3.6.3 Same-condition A/A threshold-exceedance

The same-condition A/A analysis compares two estimates formed from the same measured condition under the same phase policy and reporting rule. No intentional change in workload, implementation, framework, hardware, measurement condition, phase policy, or reporting rule separates the two estimates. Any difference between them is therefore interpreted as variation produced by the measurement and summarization procedure under the specified condition.

For the block-preserving A/A analysis, two distinct eligible measurement blocks b_*A* and b_*B* are selected from the same measured condition c. Eligible block pairs are all unordered pairs of distinct measurement blocks with at least k eligible timing observations, where k is the reporting-rule window size. For each Monte Carlo draw, one eligible block pair is sampled uniformly, and one contiguous ordered window of size k is sampled uniformly from all possible windows within each selected block. Windows may overlap across Monte Carlo draws, but each comparison pair uses one window from each of two distinct measurement blocks. Conditions with fewer than two eligible measurement blocks or fewer than k eligible timing observations per block are excluded for that reporting rule and recorded in coverage diagnostics. The reporting rule then produces estimates θ^A and θ^B. The relative difference is

Drel=|θ^A-θ^B|(|θ^A|+|θ^B|)/2.(6)

For a practical relative-difference threshold τ, the same-condition threshold-exceedance rate is

q^c,m,p,r(τ)=1M∑j=1M𝕀[Drel,j>τ],(7)

where M is the number of Monte Carlo draws or enumerated comparisons, m is the metric, p is the phase policy, and r is the reporting rule. This quantity estimates how often the same measured condition can produce a relative difference above τ under the specified phase policy and reporting rule.

Unless otherwise stated, each measured condition, metric, phase policy, reporting rule, and analysis variant used M = 5000 Monte Carlo comparisons. Monte Carlo sampling used seed 42, with condition-specific random streams derived deterministically from the condition identifiers, metric, phase policy, reporting rule, and analysis variant so that the analysis is reproducible given the same inputs.

The threshold-exceedance rate should be interpreted as a condition-specific diagnostic. It shows how often the recorded measurement process, phase policy, block structure, metric, threshold, and reporting rule produced a difference above the chosen threshold when no intentional change was made. It should not be read as a formal Type I error rate or as a general probability that a future efficiency claim is wrong. Thresholds of 5%, 10%, and 20% are used as sensitivity levels for practical interpretation; they are not proposed as universal standards. For M = 5, 000, the binomial Monte Carlo standard-error scale for an exceedance proportion is at most about 0.71 percentage points. The Results therefore emphasize ranges across execution contexts and sensitivity to block structure rather than Monte Carlo sampling error. Conventional summaries such as means, standard deviations or coefficients of variation, medians, and ranges characterize the distribution and dispersion of the measurements. The same-condition threshold-exceedance rate addresses a different reporting question: how often two estimates formed under the same measured condition differ by more than a specified practical relative-difference threshold.

3.6.4 Block structure and sensitivity analyses

The block-preserving A/A analysis keeps the recorded measurement blocks intact. It compares one window of observations from one measurement block with one window from a different measurement block of the same measured condition. This matches the situation in which two efficiency estimates are formed from separate sessions or repeated executions. It also avoids assuming that all accepted timing observations are freely interchangeable when measurement order or session-level effects may matter.

Three sensitivity analyses are used to show how much the result depends on this block-preserving choice. The adjacent-block analysis compares neighboring measurement blocks in the order in which the blocks started. The within-execution session-pair analysis compares different session blocks from the same repeated execution. The random-split analysis ignores the recorded block structure, randomly divides accepted timing observations from the same measured condition into comparison groups, and applies the same reporting rules. Comparing these analyses shows whether averaging appears more effective only when the recorded block structure is ignored.

Additional diagnostics were used to describe ordering effects and runtime associations within the measurement record. Within each measurement block, wall-clock time was screened for monotonic trends using Kendall's τ between measurement order and wall-clock time, with p < 0.05 treated as a descriptive flag. Lag-1 autocorrelation was screened using |r1|>1.96/n, where n is the number of eligible timing observations in the block. Runtime-covariate diagnostics described associations between wall-clock time and recorded runtime state, including pre-execution CPU utilization, memory state, context-switch deltas, process I/O deltas, and related process counters where available. These diagnostics contextualized timing variation but were not used to compute the block-preserving A/A threshold-exceedance estimate.

3.6.5 Estimated observation counts for target differences

This analysis estimates how many timing observations per group would be needed for a target relative difference under the observed variability of the selected metric. The calculation is used as a planning index: it asks how many observations per group would be needed for a difference such as 10% under an equal-size two-sample normal approximation.

For standard deviation s, per-group observation count n, significance level α, and target power 1 − β, the approximate minimum detectable difference is

MDD=(z1-α/2+z1-β)s2n.(8)

For a target relative difference r, the corresponding absolute difference is δ=rx̄, where x̄ is the measured-condition mean. The approximate required sample size per group is

nreq=2(z1-α/2+z1-β)2(sδ)2.(9)

Unless otherwise stated, the sample-size summaries use a two-sided α = 0.05, target power 1 − β = 0.80, and a target relative difference of r = 0.10. For wall-clock summaries under the configured phase policy, n_obs is the number of eligible timing observations available for each comparison group.

In the controlled condition, the configured phase policy excludes the first five accepted timing observations. A complete session block therefore contributes 35 eligible timing observations from the original target of 40. When the three session blocks from one repeated execution are combined, the eligible count is typically 105. If a run configuration did not reach the full target, the observed eligible count is used instead.

These calculations provide rough guidance on the number of observations needed for a target difference. Because the timing records may include order dependence, session structure, execution-context effects, and non-ideal distributional behavior, the resulting sample-size values should be interpreted as approximate rather than exact.

3.6.6 Workflow for interpreting claimed differences

Table 4 summarizes how the measurement record is used to interpret a claimed CPU-only efficiency difference. The claim is easier to support when repeated measurements of the same condition rarely produce differences above the practical relative-difference threshold, and when enough timing observations are available for the difference being claimed. Support is weaker when threshold-exceedance is common, or when the available timing observations are too few to support that difference. These labels guide how the result should be worded; they are not formal decision thresholds.

Table 4. Workflow for applying the same-condition A/A decision-risk audit to CPU-only machine-learning efficiency measurement.

The purpose is to compare a claimed difference with variation observed under the same measured condition, phase policy, and reporting rule. A comparative CPU-only result should therefore state the observed timing or derived-rate difference together with the execution context, measurement condition, phase policy, reporting rule, number of accepted timing observations, measurement-block structure, practical relative-difference threshold, same-condition threshold-exceedance rate, and estimated observation counts used to support the interpretation.

4 Results

The Results section begins by checking whether the measurement record is complete and whether the recorded functional-output summaries remain consistent. It then shows how timing and derived-metric dispersion vary by workload role, execution context, and phase policy. Finally, it reports same-condition A/A threshold-exceedance and estimated observation counts for the five representative thread-count-1 workload scenarios. These results show how much repeated-measurement evidence is needed before a CPU-only efficiency difference can be interpreted with confidence.

Unless otherwise stated, values reported at the workload-role level are medians across the eligible measured conditions for that role. These medians are computed separately for the relevant measurement condition, phase policy, metric, requested thread count, and reporting rule. When variation across execution contexts is important, context-level ranges are reported separately. The block-preserving A/A summaries are limited to the five representative thread-count-1 workload scenarios.

4.1 Audit record completeness and functional-output summary consistency

Table 5 summarizes measurement coverage and functional-output summary consistency. The target count for each context–condition combination was 43, 200 accepted timing observations.

Table 5. Measurement coverage and functional-output summary consistency.

CPU-utilization gate deferrals are reported as a percentage of total recorded attempts.

Measurement coverage was nearly complete. Five of the six context–condition combinations reached the full target of 43, 200 accepted timing observations. The only shortfall occurred for Machine 1 under the controlled condition, where 43, 160 timing observations were accepted. CPU-utilization gate deferrals occurred only under the controlled condition, as expected.

For completed run configurations, each context–condition combination had a maximum of one unique normalized output-summary hash. Thus, the later timing and derived-metric differences are interpreted as measurement and runtime variability, not as differences in the recorded workload-output summaries.

4.2 Dispersion, context, and phase policy as audit inputs

Figure 2 gives the first overview of dispersion in the Results section. It shows the full-run pattern across workload roles and metrics, providing context for the A/A analyses that follow. Table 6 then gives the condensed role-level values used in the text.

Table 6. Dispersion indicators by workload role for the full-run phase.

CV denotes coefficient of variation; Ctl. and Ref. denote controlled and reference measurement conditions. Wall CV and CPU-time CV refer to wall-clock time and process CPU time, respectively. Role-specific derived CV refers to effective bandwidth for memory streaming, estimated GFLOP/s for matrix multiplication, and throughput for framework workloads.

For wall-clock time, controlled-condition CVs for memory stream, NumPy matrix, and PyTorch inference ranged from 22.7% to 23.4%, while reference-condition values ranged from 21.1% to 22.8%. TensorFlow inference had higher wall-clock CVs of 67.1% and 58.7% under the controlled and reference conditions, respectively. TensorFlow training had the largest wall-clock dispersion, with CVs of 133.9% and 134.7%. These dispersion summaries define the variability context in which claimed CPU-only efficiency differences must be interpreted.

The derived metrics showed the same pattern. Under the controlled condition, role-specific derived CVs were 17.3% for memory bandwidth, 20.8% for NumPy estimated GFLOP/s, 20.0% for PyTorch throughput, 21.6% for TensorFlow inference throughput, and 22.2% for TensorFlow training throughput. In other words, converting wall-clock time into throughput, estimated GFLOP/s, or effective bandwidth did not remove the timing variation; it carried that variation into the derived metric.

Dispersion was also context-dependent (Figure 3). In the controlled full-run phase, NumPy matrix had the narrowest cross-context range, with median wall-clock CV values from 19.5% to 26.9%. PyTorch inference ranged from 20.3% to 31.2%, and memory stream ranged from 15.2% to 26.9%. TensorFlow inference ranged from 37.5% to 182.7%, and TensorFlow training ranged from 73.0% to 380.3%, with the largest values observed in the Colab allocation context.

Phase policy changed the interpreted efficiency evidence for several roles. Figure 4 shows how wall-clock CV changed after excluding 0, 5, 10, or 20 initial accepted timing observations. Table 7 summarizes the corresponding measured-condition-level CV changes after role-level aggregation. NumPy matrix and PyTorch inference showed limited warm-up sensitivity, with median absolute full-to-configured-phase relative CV changes of 2.79% and 3.65%, respectively. Memory stream, TensorFlow inference, and TensorFlow training showed larger changes of 29.5%, 63.0%, and 79.7%. The corresponding configured-phase median wall-clock CVs were 22.8%, 26.4%, 26.3%, 22.6%, and 24.2%. These results show that full-run and configured-phase summaries can support different reporting claims, so they should not be treated as interchangeable.

Table 7. Controlled-condition phase sensitivity, same-condition A/A threshold-exceedance, and estimated wall-clock observation counts for the five representative thread-count-1 workload scenarios.

The CV-change column reports the median absolute relative change in wall-clock CV from the full-run phase to the configured phase policy, not an absolute percentage-point difference. A/A columns report cross-block contiguous-window comparisons at the 10% practical relative-difference threshold; values are medians across the three context-specific results. Observation-count columns use configured-phase measured-condition-level wall-clock means and standard deviations with two-sided α = 0.05, target power 1 − β = 0.80, and a 10% target relative difference. The MDD column uses the observed eligible per-group count under the configured phase policy.

4.3 Same-condition A/A threshold-exceedance

The main A/A result is the threshold-exceedance analysis. This analysis asks a simple question: when the same measured condition is summarized twice using the same reporting rule, how often do the two estimates differ by more than a practical relative-difference threshold? To answer this, the block-preserving analysis compares contiguous windows from different recorded measurement blocks. This keeps the session and repeated-execution structure intact instead of treating all accepted timing observations as freely interchangeable.

At the 10% practical relative-difference threshold, reporting based on one accepted timing observation showed high same-condition threshold-exceedance. Under the controlled condition and configured phase policy, the role-level median rates ranged from 77.6% to 80.6% across the five representative thread-count-1 workload scenarios. Mean-of-10 reporting reduced the rates, but they still remained high, ranging from 67.1% to 76.1%.

The role-level medians also hid important execution-context variation. For controlled-condition mean-of-10 reporting, threshold-exceedance ranged from 35.1% to 85.0% across the 15 context–scenario combinations. These results show that differences near common practical relative-difference thresholds need block-aware repeated-measurement evidence before they can be described as strongly supported.

The reference condition showed a similar pattern. Cross-block reporting based on one accepted timing observation produced threshold-exceedance rates from 77.9% to 80.7%. Mean-of-10 reporting reduced the rates only slightly, to 74.6%–78.2%. This comparison shows that the results are sensitive to the measurement condition. It should not be read as evidence that either the controlled or reference condition is generally better.

The sensitivity analyses led to the same conclusion. Adjacent-block comparisons and within-execution session-pair comparisons both showed high exceedance when the estimate was based on one accepted timing observation. Mean-of-10 reporting reduced threshold-exceedance, but it did not remove it.

Figure 5 shows the same pattern across the 5%, 10%, and 20% practical relative-difference thresholds. As the threshold increases, the curves generally move downward, and averaging also lowers the curves. However, threshold-exceedance remains substantial across workload roles.

Random-split analyses produced lower mean-of-10 rates: 42.8%–55.8% under the controlled condition and 44.2%–66.6% under the reference condition. This gap shows why the recorded measurement-block structure matters. Averaging appears more effective when the block structure is ignored than when the analysis preserves the structure that a reported comparison may inherit.

4.4 Claim-support matrix and worked example

Table 8 summarizes the evidence used to interpret a claimed 10% CPU-only wall-clock difference under mean-of-10 reporting. Across the five representative workload scenarios, same-condition A/A threshold-exceedance remained high, ranging from 67.1% to 76.1%. The observation-count and MDD values show that a 10% difference is near the estimated measurement boundary for these summaries. However, because same-condition threshold-exceedance remains common under block-preserving comparisons, mean-of-10 reporting alone does not provide strong support for a claimed 10% wall-clock difference.

Table 8. Evidence summary for interpreting a claimed 10% CPU-only wall-clock difference under the controlled condition and configured phase policy.

Table 9 gives one worked example of how the audit evidence changes the wording of an efficiency claim. The example considers a hypothetical 10% wall-clock improvement for the representative PyTorch inference scenario under the controlled condition and configured phase policy, using mean-of-10 reporting. It illustrates the interpretation for one workload scenario and reporting rule; it is not a general conclusion about PyTorch inference.

Table 9. Worked audit example for a hypothetical 10% CPU-only wall-clock improvement.

The example shows how same-condition evidence changes the wording of an efficiency statement. Under this audit setting and reporting rule, a 10% point estimate should be described cautiously because similar-sized differences often arise from repeated measurements of the same measured condition.

4.5 Estimated observation counts and measurement-procedure diagnostics

The sample-size estimates show how many timing observations per group would be needed to support a 10% target relative difference under the observed wall-clock variability. Table 7 reports the configured-phase values used to interpret claimed differences. Across workload roles, the median estimate ranged from 81 to 110 observations per group, and the corresponding MDD values ranged from 8.75% to 10.2%.

The eligible observation count is close to this range. Under the configured phase policy, each complete session block contributes 35 eligible timing observations after excluding the first five accepted timing observations. Combining the three session blocks within one repeated execution therefore typically gives n = 105 eligible timing observations.

Figure 6 shows that the estimated observation count depends on the phase policy. In the full-run phase, the required observation counts were much larger for the TensorFlow roles: the median requirement was 909 observations per group for TensorFlow inference and 3,120 observations per group for TensorFlow training. These large counts show why differences near 10% need explicit repetition evidence and a clearly stated phase policy.

The measurement-procedure diagnostics supported the block-aware interpretation. The CPU-utilization gate sensitivity runs compared four settings under the controlled condition: no gate, 60%, 80%, and 95%. Changing the gate threshold affected how often observations were deferred. Median per-run deferment was 0% with no gate, 6.98% at the 60% threshold, 2.44% at the 80% threshold, and 0% at the 95% threshold.

However, changing the CPU-utilization gate did not remove workload-role-dependent timing dispersion. Across gate settings, median wall-clock CVs stayed in similar ranges for memory stream (23.1%–23.7%), NumPy matrix (22.0%–23.5%), and PyTorch inference (23.6%–27.2%). TensorFlow inference (64.6%–71.6%) and TensorFlow training (134.9%–142.9%) remained higher. The CPU-utilization gate is therefore treated as part of the measurement condition, not as a general solution for eliminating timing dispersion.

Ordering diagnostics also showed that accepted timing observations should not be treated as freely exchangeable. In controlled full-run wall-clock summaries, trend screens were flagged in 33.3%–47.8% of workload-role summaries, and lag-1 autocorrelation screens were flagged in 24.2%–65.2%. Under the configured phase policy, lag-1 autocorrelation screens were still present in 47.4%–60.0% of workload-role summaries. These results support the block-aware interpretation used in the same-condition A/A analysis.

The Results lead to four main findings. First, the measurement record separated functional-output summary consistency from timing dispersion. Second, workload role, execution context, and phase policy all changed the variability context used to interpret CPU-only efficiency differences. Third, same-condition A/A comparisons often exceeded practical relative-difference thresholds, especially when the analysis preserved the measurement-block structure. Fourth, differences near 10% should not be interpreted from isolated timing or derived-rate summaries alone; they require explicit repetition, phase-policy reporting, and block-aware evidence.

5 Discussion

The central implication of the results is that CPU-only efficiency claims should be treated as measurement-supported claims, not as conclusions that follow directly from isolated timing or derived-rate values. A 10% timing or throughput difference may look meaningful when reported as a point estimate, but the same measured condition can also produce differences of similar size when it is repeated and summarized in the same way. The audit therefore shifts the interpretation from the observed size of a timing or derived-rate difference to the strength of the evidence behind it, evaluated against same-condition measurement variation under the matching phase policy and reporting rule.

This matters because a timing value alone does not define the evidence behind an efficiency claim. The interpretation also depends on the measured condition, the phase policy, the reporting rule, and the measurement-block structure used to form the estimate. In the present results, preserving measurement blocks led to a more cautious interpretation than random splitting. This means that accepted timing observations should not be treated as one homogeneous pool simply because they share the same measured condition. When a comparison is based on observations collected in separate session blocks or repeated executions, the support for the claim should be evaluated using that same recorded structure.

The distinction between functional-output summary consistency and timing dispersion is also important for interpreting CPU-only experiments. Consistent output-summary hashes make the measurement record more auditable, but they do not by themselves make the timing evidence strong. The useful interpretation is therefore not that deterministic outputs are unimportant, but that they answer only one part of the measurement question. They support consistency of the recorded workload-output summaries, while same-condition A/A threshold-exceedance and estimated observation counts address the strength of the timing evidence for an efficiency claim.

The phase-policy results show that a measurement record can support different claims depending on which observations are included. A full-run summary and a configured-phase summary are not interchangeable versions of the same result; they describe different subsets of the measurement record. This matters when early observations, warm-up behavior, or session effects change the dispersion pattern. For this reason, phase policy should be treated as part of the claim definition rather than as a minor reporting detail.

The audit can be used as a practical check before making a strong CPU-only efficiency claim. The key question is whether repeated measurements of the same documented condition often produce a difference as large as the claimed one when the same reporting rule is used. If same-condition threshold-exceedance is high, the claim should be worded more cautiously, supported with more block-aware repetition, or described as conditional on the measurement setting. If threshold-exceedance is low and the available timing observations meet the estimated observation count, the same evidence can support a stronger efficiency statement.

5.1 Threats to validity

The controlled and reference conditions differed in bundled design elements, including CPU-utilization gate filtering, configured warm-up exclusion, inter-session delay, and post-measurement pacing. Because these elements changed together, the comparison describes the behavior of two documented measurement conditions. Identifying the independent effect of each design element would require additional targeted experiments.

The CPU-only controls documented the intended execution condition, but residual timing variation could still arise from CPU-frequency changes, scheduler preemption, incomplete hardware isolation, native-library thread behavior, framework-internal scheduling, memory pressure, and operating-system activity. These limits mean that the numerical findings should be interpreted within the measured execution contexts, workload scenarios, measurement conditions, phase policies, and reporting rules used in this study.

6 Conclusions

This paper introduced a same-condition decision-risk audit that evaluates whether reported CPU-only machine-learning efficiency differences are supported by repeated-measurement evidence rather than by point estimates alone. Across eight workload scenarios and three execution contexts, the audit showed that consistent functional-output summaries can coexist with high timing and derived-metric dispersion, and that block-preserving same-condition comparisons frequently exceeded practical relative-difference thresholds even when averaging multiple observations. The results demonstrate that phase policy, measurement-block structure, and execution context must be treated as integral parts of any efficiency claim, and that differences near common practical relative-difference thresholds need explicit sample-size context for strong support. We recommend that future CPU-only efficiency studies, and other timing-based performance studies with comparable repeated-measurement structures, routinely report the measurement context, phase policy, reporting rule, block structure, same-condition threshold-exceedance rate, and estimated observation counts so that readers can judge the evidence behind a claimed difference.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI was used only forlanguage editing, clarity improvement, formatting suggestions, and assistance with manuscript organization.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

Edited by: Hongji Yang, University of Leicester, United Kingdom

Reviewed by: Jay Lofstead, Sandia National Laboratories (DOE), United States

Robin Boëzennec, Université de Lyon, France