Quality control in multiple-breath washout measurements: an automated, model-based approach quantifies effects on primary outcomes

Quality control (QC) is an important but challenging step in the correct interpretation of multiple-breath washout (MBW) tests, as irregular behaviors are frequent, particularly among young and ill patients, and can lead to biased outcomes. We developed a model-based, automated QC algorithm to estimate the effect of irregular behaviors on MBW outcomes. It compares results between simulated measurements containing either ideal target behaviors or the observed irregular behaviors of the measurement. Differences in primary outcomes - lung clearance index (LCI), functional residual capacity - between simulations served as behavior-specific QC outcomes. We validated the automated QC algorithm against a dataset of 2,471 measurements of 87 children with cystic fibrosis (CF). We compared the results of automated QC with the ones performed by experienced raters in a second dataset of 100 measurements of healthy children and children with CF. The test could be applied successfully to all measurements. We found that the impact of QC-related factors on MBW outcomes explained around 45% of the within-visit variability of LCI in children with CF. The largest effect on LCI was caused by leaks or trapped gas (in 31% of measurements), changes in end-expiratory lung volume (23%), and variations in breath size around the end-of-test (19%). For the second dataset, the agreement between automated QC and QC by experienced raters was moderate but comparable to QC performed by two experienced raters. Our novel automated QC tool estimates the accuracy of MBW outcomes, providing a fast, reproducible and impact-based method to perform QC for MBW measurements.

Multiple-breath washout (MBW) is an established technique to assess ventilation inhomogeneity in the lungs (Gustafsson, Aurora, and Lindblad, 2003; Gustafsson et al., 2008; Jensen et al., 2013; Owens et al., 2011). In nitrogen (N 2) MBW, subjects wash out the resident N 2 in their lungs using pure oxygen (O 2). The test’s main outcomes are the lung clearance index (LCI) and the functional residual capacity (FRC). MBW tests typically last several minutes, and while they mostly require only passive cooperation from the test subject, there are irregular behaviors (e.g., irregular or non-tidal breathing, coughs, leaks), which can lead to biased or unusable test outcomes (Robinson et al., 2013; Yammine et al., 2016; Robinson et al., 2018; Yammine et al., 2014). A quality control (QC) step, which identifies such irregular behaviors, is therefore an important component of the correct interpretation of MBW measurements. The application of QC has been shown to have a significant impact on the variability of MBW outcomes, particularly on repeatability of LCI (Frauchiger et al., 2023; Oude Engberink et al., 2017; Wee et al., 2025; O'Neill et al., 2018).

There are existing QC guidelines for MBW, which allow experts to make judgments (expert QC) on whether they should include or exclude measurements for research and clinical routine (Frauchiger et al., 2021; Jensen et al., 2016; Saunders et al., 2020). These guidelines allow expert raters (typically trained physicians, nurses or research scientists) to assess whether the test subject behaved in a way which led to an adverse effect on the test’s main outcomes, independent of the ventilation inhomogeneity that is being measured.

Performing this expert QC on MBW data represents the current gold standard, but it has several major limitations: Expert QC is very time-consuming and the level of expertise required for it is not common in clinical routine. This represents a barrier for more widespread use of MBW.

In cases where the irregular behavior is of intermediate quality - between clearly acceptable and clearly unusable - expert QC has a significant subjective component that might lead some raters to accept and others to reject a given measurement. This is especially the case when there are multiple instances of minor irregularities within one measurement. As MBW measurements are typically several minutes long, and as they are also performed in awake people starting in pre-school age, it is very common for them to contain such “grey zone” irregular breathing, especially if a lung pathology is involved. This subjectivity of QC, outside of dedicated central over-reading centers for MBW data, makes consistent QC difficult and potentially leads to non-reproducible results with a difficult-to-quantify bias.

While it is feasible to spot irregularities in the signals of a MBW measurement, it is very challenging to estimate the impact or importance of such irregularities on LCI or FRC. Expert QC therefore relies on empirically determined thresholds for allowed variations in behavior for each individual test.

There is therefore a need for tools to automatically perform MBW-QC in an objective, reproducible manner.

Study design

This is a proof-of-concept study to first, develop a novel QC tool, and second, assess its practicability using existing datasets of healthy children and children with cystic fibrosis (CF). Written informed consent was obtained from patients and caregivers and the study was approved by the ethics committee of the Canton of Bern (2019-01072, 2017-02139).

Development of the QC tool

Objective QC

In order to develop an objective automated QC algorithm, we aimed to relate potentially problematic breathing irregularity events (e.g., coughs, non-tidal breathing, leaks) directly with estimated changes in MBW outcomes. We therefore developed a model, which - for any given breath - related an input of breathing behavior with an output of expected N 2 dilution. To do this, it took into account the current end-expiratory lung volume (EELV), physiological and machine dead spaces, and how the N 2 washout efficiency changed over the course of washout for a given test subject.

Using this model, we ran washout simulations to make side-by-side comparisons between simulated measurements with a given observed irregularity (e.g., the observed changes in EELV during a measurement) and simulated measurements without said irregularity (i.e., if EELV had remained constant). If the two simulations finished with similar outcomes, we could conclude that the irregularities in question were minor enough not to affect main outcomes. Conversely, if the simulations showed a significant discrepancy in outcomes, we could conclude that the irregularity in question was expected to have a significant effect on MBW outcomes in the original measurement.

MBW model

The washout model we used to simulate measurements consisted of the following elements: A mixing alveolar volume (V A), a non-mixing volume of dead space (V D), and a decreasing dilution efficiency as a function of N 2 washed out of the lungs (Figure 1). The input to this model was a series of ingoing and outgoing breath volumes V i n s p, V exp. The dilution of N 2 was modelled to take place in the ventilated volume V A.

Before inspiration, the N 2 was contained in the volume V A + V D, and after inspiration it was contained in volume V A + V D + V i n s p – V D = V A + V i n s p. Therefore, the dilution factor K was equal to and is dimensionless as a ratio of volumes. The alveolar volume V A − n e w after expiration was adjusted by the volume difference between inspired and expired volume,

This dilution was modelled to be modified by an efficiency factor (f), which either stayed constant or decreased over the course of the washout as a function of how much N 2 was left in the lungs (Supplementary Figure S1). The concentration N 2 − n e w [volume fraction] after expiration was given by where K is defined as in Equation 1. The changing efficiency of this dilution simulated the fact that well-ventilated compartments initially wash out more efficiently in MBW, and that over time most of the remaining N 2 is situated in less well-ventilated parts of the lungs, slowing down the washout process (Gustafsson et al., 2014). An efficiency factor f = 1 in the washout model corresponded to homogeneously ventilated lungs.

We defined the washout progress of a breath (P b r e a t h) as the difference between normalized logarithmic end-tidal concentrations (Supplementary Equation S2) of N 2 between two breaths. Expressing washout progress this way allowed a more direct comparison between dilutions throughout the washout. A breath of the same size and efficiency leads to different changes in absolute end-tidal concentrations at the start and end of washout. However, the same dilution of N 2 occurs, and therefore the same progress P b r e a t h. Normalization was chosen such that the cumulative washout progress (P c u m u l a t i v e) required to complete an MBW measurement was equal to 1. Consequently, ten breaths with P b r e a t h = 0.1 were expected to finish a measurement, equivalent to five breaths with P b r e a t h = 0.2. On the logarithmic scale, the washout efficiency f of a breath was the ratio between the measured progress of a breath and the progress predicted by the MBW model assuming homogenously ventilated lungs (Supplementary Equation S1).

The parameters used by the MBW model (V A, V D, and efficiency factor P c u m u l a t i v e) were estimated from each MBW measurement individually. V D could be estimated from the CO 2 and flow signals (by measuring at which expired volume there was a rise in CO 2, see Supplementary Material Section 1.3), and V A was equal to FRC minus V D. Efficiency was first calculated per breath by comparing the observed dilution of N 2 with the predicted dilution in homogenously ventilated lungs. Second, the resulting efficiency values per breath were used to determine a linear model of efficiency as a function of cumulative washout progress, excluding outliers (Supplementary Figure S1). This linear fit could be used to determine the efficiency factor for a given breath as a function of the washout progress.

Quality control tests

Current QC guidelines (Frauchiger et al., 2021a; Jensen et al., 2016; Saunders et al., 2020) focus on identifying behaviors which are known to impact LCI and/or make it less meaningful. These factors include: Breathing artifacts such as changes in EELV, breaths of the wrong size (especially at the beginning or end of test), hypo-or hyperventilation,

Leaks or trapped gas released throughout the measurement, leading to unexpected rises in end-tidal N 2, or

Technical issues such as breathing artifacts which might lead to early termination of the measurement, insufficient wait time between measurements and signal synchronization issues.

We developed a set of tests for automatic quality control. For each test we chose a potential source of error from the above list (either behavioral or methodological), and attempted to estimate its effect by one of two ways: Running two simulations using the washout model, where the inputs differed by the behavior in question, or

Analyzing the measurement with two differing methods (e.g., using different breath detection or signal synchronization methods).

Each test resulted in an estimated effect, defined as the difference between the results obtained with a given behavior or methodology and those without the behavior or with an alternative methodology (expressed in percentage).

These tests, together with simple numerical tests for initial N 2 concentration and inspiratory overflow, were chosen to cover the same irregular behaviors and sources of error as those examined in current expert QC procedures. All tests can be evaluated in an automated, reproducible manner. The specific tests are listed below.

Test 1: End-expiratory lung volume (EELV)

In order to test for the effect of changing EELV, we compared the MBW outcomes of i) washout simulations using the measured V i n s p and V exp of the measurement as input, vs. ii) washout simulations using the measured V i n s p but setting V exp = V i n s p, which leads to a stable EELV over the course of the measurement.

Test 2: Tidal volume (VT) variability

To test for the effect of variable V T, we compared i) washout simulations using measured V i n s p, but where V exp = V i n s p (resulting in stable EELV but variable V T), vs. ii) washout simulations where all V i n s p and V exp were equal to the median V T of the measurement (resulting in stable EELV and stable V T).

Test 3: VT size

To test for the average effect of the measured V T size, we compared i) washout simulations where all V i n s p and V exp were equal to the median V T of the measurement vs. ii) washout simulations where all V i n s p and V exp were equal to the most efficient V T, calculated based on estimated dead-space V D and alveolar volume V A (see Supplementary Material Section 1.4).

Test 4: Leak/trapped gas

To test for the impact of unexpected upticks of N 2 concentration (due to either leaks or trapped gas released throughout the measurement), we compared i) washout simulations where the end-tidal concentration of N 2 was equal to the measured N 2 vs. ii) washout simulations where end-tidal N 2 was simulated by the washout model. As the washout model ignored major outliers when fitting efficiency to washout progress, the washout simulation provided a reference for an average washout and would not incorporate leaks.

Test 5: End-tidal concentration measurement

To test for the effect of breath size on end-tidal concentration and consequently end-of-test determination, we compared the outcomes of i) a standard MBW analysis vs. ii) a more robust MBW analysis with an improved size-independent method for determining the end-tidal concentration (see Supplementary Material Section 1.5).

Test 6: Synchronization

To test for the effect of calibration on the signal synchronization we compared the outcomes of i) a standard MBW analysis vs. ii) an MBW analysis with recalculated synchronization (see Supplementary Material Section 1.6).

Test 7: Breath detection

To test for the effect of small breaths and irregular breathing patterns which could be misinterpreted as an early termination of the measurement, we compared the outcomes of i) a standard MBW analysis vs. ii) a more robust MBW analysis with a breath detection that did not remove breaths and which had a dead-space V D -dependent breath detection threshold (Oestreich et al., 2022).

Application and validation of the QC tool

Datasets

To characterize the results of automated QC and the effect of irregular behaviors on measured within-visit variability, we re-analyzed an existing dataset of 4490 MBW measurements, originating from 1,618 longitudinal visits of 87 children with CF. The measurements were conducted between 2012 and 2019. A smaller version of this dataset has been described in detail elsewhere (Frauchiger Bettina et al., 2021). In order to make within-visit correlations of QC-related estimates with MBW outcomes, we included only visits which had at least three completed measurements.

To compare the outcomes of automated QC with QC performed by expert raters (expert QC), we randomly selected 50 completed MBW measurements each from 6 year old healthy children and 6 year old children with CF from measurements conducted between 2020–2025 in the Bern Basel Infant Lung Development (BILD) (Fuchs et al., 2012) and Swiss (Infant Development (SCILD) (Korten et al., 2018) cohorts respectively. This dataset was chosen to be a representative sample of measurements from clinical routine, in an age group most likely to produce QC-relevant behaviors.

N 2 MBW measurements were collected using the Exhalyzer® D (Ecomedics AG, Duernten, Switzerland), and analyzed using Spiroware 3.3.1.

Descriptive application

To characterize the application of the QC tool to the real datasets of healthy children and children with cystic fibrosis, we determined feasibility, time requirements, and the QC tests with the largest and most frequent impact on MBW outcomes.

Association to LCI

Each of the quality control tests provided an estimate of the impact of individual irregularities on MBW outcomes. Most tests estimated an effect on cumulative expired volume (CEV), i.e., the length of the test, while others (synchronization, leaks) estimated both an effect on CEV as well as on FRC. To obtain a single impact estimate of the effect on LCI, we multiplied the individual estimated effects on CEV originating from the different QC tests, and divided this by the effects on FRC.

In order to compare within-visit variability of LCI with the QC-based impact estimate across visits and individuals, we divided both the LCI and estimated impact on LCI by the mean value of the given visit.

Comparison to expert QC

The automated QC algorithm produced an estimate for the impact of QC-related factors on MBW outcomes. In order to obtain a single binary outcome of acceptability per measurement, a threshold of acceptable estimated impact had to be defined. This threshold can be adapted depending on the severity of QC desired. We defined measurements to have acceptable QC if no individual test had a higher QC-related change in MBW outcomes than 15%, and the numerical thresholds (e.g., for initial concentration) were passed.

In order to compare the outcomes of automated QC with expert QC, we compared the binary outcome of “accepted/rejected” from the automated QC with the original QC performed by the expert rater conducting the measurement, as well as a second round of retrospective QC performed by a different experienced expert rater.

Statistics

The effects of individual quality control tests were quantified using descriptive statistics by computing the fraction of samples that exceeded the threshold and the fraction of times one test produced the highest result. To evaluate the association between relative LCI and aggregated or individual QC impact estimates, we computed coefficients of determination (R 2). The agreement of within-visit variability of LCI with estimated QC impact was measured by computing the Pearson correlation coefficient for the relative measurement outcomes of each visit. To compare automated QC with one of the two expert QC, we computed Cohen’s kappa between each pair of the three instances of QC.

Statistics were performed using MATLAB R2022a (The MathWorks Inc., Natick, Massachusetts, United States), and Python 3.12 (Python Software Foundation).

Datasets

In the large dataset of 87 children with CF (dataset 1), a total of 2,471 measurements from 754 visits by 84 participants had at least three measurements per visit, and were included in the analysis (Table 1).

The smaller dataset of randomly selected measurements from 6-year-old children included 50 measurements from 22 children with CF, and 50 measurements from 40 healthy children (Table 1).

Objective QC

It was possible to apply the quality control algorithm to all measurements in a fully automated manner, without any manual intervention. Evaluation time of a single measurement (including MBW analysis) on an average office computer (HP EliteBook 8, CPU Intel Ultra 5, 16 GB RAM) was on the order of 4s using the current non-optimized implementation of the automated QC code, which translates to 7 min for 100 files.

Quality control tests

An illustrative example of the automated QC method using the lung model is shown in Figure 2 for the EELV test. The left and right columns illustrate measurements of the same visit. Each column shows the comparison between a simulation using the measured breathing volumes of a measurement (including breathing irregularities, here specifically changing EELV) and the resulting washout progress, compared to one without EELV irregularity, in two different MBW measurements. For the two cases the washout progress differs resulting in different measurement end points and thus MBW outcomes.

End-expiratory lung volume, end-tidal concentration measurement, and unusual rises in expired N 2 concentration (leaks, trapped gas) were the tests which most often predicted the largest effect, and were most frequently above the threshold of an estimated impact on LCI of 15% (Table 2).

The overall QC acceptance ratio (the proportion of visits which had no test which had a higher estimated effect than 15%) was 76%. The mean per-visit variance of LCI was 0.72 TO when including all measurements, and reduced to 0.39 TO when only including measurements accepted by the automated QC algorithm.

The QC acceptance ratio per age group is shown in Supplementary Table S1, confirming that QC-related outcomes improve as the children age and become more adept at following the instructions around a MBW measurement.

Association to LCI

There is a significant correlation of the within-visit variation of LCI and the impact estimate by the quality control algorithm. The combined estimate of all quality control tests for the impact on LCI managed to explain a significant proportion of the within-visit variability between individual MBW tests (R 2 = 0.45) (Figure 3; Table 3).

The individual tests did not all contribute equally to the correlation with within-visit variability, with the largest share of the correlation explained by the effect of EELV, followed by end-of-test determination via end-tidal concentration, leaks and release of trapped gas, and tidal volume size, in that order. Finally, the correlation of the technical issues related to synchronization, breath detection, and inspiratory overflow leaks, as well as tidal volume variability with within-visit variability was minor (Table 3).

The association of measurements of individual visits are shown in Supplementary Figure S3. The median Pearson correlation coefficient of within-visit LCI with estimated effects of QC-related factors was 0.83 (Supplementary Figure S4).

Relation to expert rating

The agreement between the automated QC and expert QC was only moderate, with a Cohen’s kappa ranging from 0.32 between the automated QC and the expert rating on the day of the test, to 0.46 between the automated QC and the retrospective expert QC. The agreement between the two instances of expert QC was higher with a Cohen’s kappa of 0.65 (Supplementary Table S2).

The association of the critical variable for automated QC (highest results of estimated impact over all tests) with the grading from expert QC is shown in Figure 4A. Figures 4B,C illustrate the disagreement between automated and expert QC for two measurements.

Summary

We developed a novel, fully automated QC tool for MBW measurements, which gauges the relevance of behavioral irregularities by estimating the direct impact of those irregularities on MBW outcomes. We applied this algorithm to datasets of healthy children and children with CF, and showed that it could explain a significant part of existing within-visit variability in MBW outcomes. Further, we characterized the differences between it and the current gold standard expert QC. The development of a fast, automated, reproducible, and standardized approach to MBW QC represents a significant advance for MBW as a measurement technique.

Discussion of results

The automated QC approach described here provides the first estimate of the impact of individual quality control issues on MBW outcomes. Rather than focusing on a binary acceptance/rejection decision, the approach estimates the accuracy of MBW outcomes with respect to irregular behavior. This allows for rapid identification of problematic behaviors within MBW measurements, and allows the relevance of different sources of error to be judged according to the same comparable scale, namely, impact on MBW outcomes.

The examples shown in Figure 2 illustrate the potential difficulty to visually judge the impact of factors such as variable EELV on MBW outcomes, as they can be transient, quickly reverse, or have different magnitudes of impact when they occur early or late in the measurement. The model-based approach used here can quantify the expected impact of these behavioral irregularities in a reproducible manner across measurements.

We could show that a significant part of the within-visit variability of MBW measurements in a large dataset of children with CF was due to factors linked to QC that could be estimated using this new algorithm. The highest correlated individual factors within this aggregated estimate were changing EELV and the breath-size-dependent determination of end-expiratory N 2 concentration. The QC-related factor which most frequently exceeded the threshold of acceptability in a dataset of children with CF was trapped gas and leaks, related to end-expiratory N 2 concentrations that deviated from washout-model-based expectations. This is likely due to MBW measurements in children with CF often showing extended washout “tails”, where the end-tidal concentration is slowly dropping below the end-of-test threshold, but then occasionally slightly increases again as trapped gas is released. While we used a fixed 15% as the threshold of acceptability for individual tests here, it might make sense in certain cases to instead relate the estimated impact to clinically relevant thresholds. E.g., if an effect of >15% is estimated due to unexpected washout progression, the LCI might be so clearly above the upper limit of normal that the estimate does not change the clinical outcome of the test. In general, an advantage of the approach described here is that QC can be automatically performed according to the needs of the study or use case. Some applications might have different objectives that require more or less strict QC based on the individuals at hand, or ignore certain specific tests according to the question.

Comparison to expert QC

We could show that the results of MBW QC showed moderate overlap to those obtained by gold standard expert QC. While the overlap between automated and expert QC, especially within the group of rejected measurements, was not perfect, this overlap was not much lower than between two separate instances of expert QC. This is likely due to several key differences between the approach described here and the current expert QC gold standard. The current standard, e.g., states that any leak (no matter the size) should lead to exclusion of the measurement. Similar to how other breathing artifacts exist on a «spectrum of acceptability», the automated QC allows for small potential leaks, as long as the estimated impact on MBW outcomes remains small. Additionally, the current gold standard defines limits of acceptable deviation from the ideal behavior for each QC test individually, while in the automated QC they are replaced by the same scale based on the impact on main MBW outcomes. Figure 4 shows two examples of measurements where the automated QC and current gold standards disagree in their final assessment of acceptability. In one case, expert QC may have struggled to visually identify whether an apparent change in EELV was sufficient to influence outcomes. In the other, there is clearly visible evidence of a small leak, which expert QC is instructed to reject, but which the automated QC identified as having little effect on outcomes.

Additional research is needed to explore how best to combine the insights from this manuscript with existing guidelines for expert QC. For example, variation in EELV was a major source of error in our datasets, but remains a very challenging factor to judge in expert QC. This suggests the need for improved methods and visualizations of relevant signals to aid experts in performing QC.

Clear advantages of automated QC are that it saves time, provides a single, repeatable standard for QC across data from different centers, and that it can be retrospectively applied to data when it is re-analyzed according to the newest MBW algorithms. Furthermore, a major difficulty in expert QC is to determine whether an anomalous signal (such as an uptick in end-tidal N 2) is due to a severe QC factor such as a leak, or whether the signals can be entirely explained by the underlying breathing behavior. The model-based approach used here inherently distinguishes between these cases, which is particularly useful in cases where the test subject is unable to attain a high quality of tidal breathing.

Comparison with other studies

At the core of the approach described here lies the washout model. In a previous study, a simpler model of homogeneously ventilated lungs using constant volumes was employed to quantify the impact of the size of tidal volume relative to FRC on LCI (Schmalisch et al., 2015). The more general model described here allows to quantify irregular breathing behavior, e.g., changes in EELV, which have a major effect on measured LCI variations. Additionally, it models ventilation efficiency.

A two compartment model of washout using a superposition of two linear components was described in a previous study (Gustafsson et al., 2014), which allows for an analysis of slow and fast ventilating compartments in MBW. The use of a linearly decreasing washout efficiency as a function of washout progress used here is a similar concept (leading to a decreasing speed of washout as the washout progresses), though the precise implementation differs.

Previous research has demonstrated that suboptimal data processing by proprietary MBW device software may occur (Wyler et al., 2024; Wyler et al., 2021), potentially affecting the accuracy of measured LCI values (Oestreich et al., 2024; Oestreich et al. 2022; Oestreich et al. 2021; Zwitserloot et al., 2020). Included in the automated algorithm are therefore alternate methods for signal synchronization, breath detection, and end-tidal concentration calculation. These are not strictly related to detecting irregular behaviors of the test subject, but instead aim to detect behaviors where the choice of processing methodology might have a major impact.

Strengths and limitations

We present a novel framework for interpreting MBW and performing quality control in an automated and objective manner. We examined the use of these new tools in a large dataset of children with CF, and compared its application to expert QC in a dataset of heathy controls and children with CF.

The algorithm described here was developed for and applied to N 2 MBW data measured with the Ecomedics Exhalyzer® D. Specifically, the automated QC relies on the presence of a CO 2 signal, a tracer gas signal, and a flow signal. Some tests, such as the synchronization and inspiratory overflow tests, are specific to the Exhalyzer® D device setup. However, only minor adaptations would be required in order to extend the algorithm to SF6MBW measurements, and, depending on the device setups, measurements originating from other devices.

A desirable time-saving feature of MBW QC would be live feedback for people as they perform the test, informing them when a behavior is likely to have compromised a test and a restart would be advisable. With the current implementation of the automated QC, the earliest time to perform the QC is at the end of the measurement. Further development would be needed to make automated QC during the measurement possible.

Conclusion

This work introduces a novel, innovative approach to automated quality control for MBW. It demonstrates a feasible method to assign estimates of impact to behavioral irregularities in MBW measurements in a fast, objective and reproducible manner, which permits fully automated QC but especially facilitates the QC of measurements with intermediate quality, which are difficult to rate by visual inspection. Furthermore, it provides novel tools with which to interpret MBW, and provides new insights into the thresholds of acceptability of MBW measurements.