Do large language models keep the hedge? Uncertainty preservation when large language models summarize aviation and maritime incident reports

1 Introduction

Reporting systems in aviation and shipping run on frank, hedged description. When a pilot files a report with the National Aeronautics and Space Administration's Aviation Safety Reporting System (ASRS) or an investigator writes up a marine casualty for the UK Marine Accident Investigation Branch (MAIB), the text's value lies partly in what it asserts and partly in how firmly it asserts it. A reporter writes that a crew “may not have seen” a vessel, that a gauge was “possibly” misread, or that a cause “could not be identified.” These are not decorations. They are the reporter's calibration of the evidence, letting a later reader separate what happened from what might have happened. The reporting systems exist precisely to gather candid, often uncertain, firsthand accounts, and their utility depends on that candor surviving into whatever record analysts later consult.

Commitment and source-of-evidence marking is a central function of language. In written discourse, it has been described using the categories of hedges and boosters, the devices by which a writer withholds or invests certainty (Hyland, 2005), and through the grammar of epistemic modality, the modal verbs and adverbs that grade how sure a proposition is Palmer (2001). It is also a matter of pragmatics: Under the cooperative principle, a careful speaker asserts no more than the evidence warrants, so a hedge signals that the speaker is honoring the maxim of quality (Grice, 1975). In a safety report, these signals are load-bearing. The difference between “the crew did not see the vessel” and “the crew may not have seen the vessel” is the difference between a finding and a hypothesis, and investigators, regulators, and, increasingly, automated systems read the record accordingly.

It is worth separating the two easily conflated questions because only one is at issue here. The first is whether a model calibrates its own confidence, that is, whether the certainty a model expresses about a claim it originates matches the probability that the claim is true. The second is whether a model, acting as a re-transmitter of someone else's claims, preserves the certainty that another writer expressed. The first is a calibration question and has a large literature; the second, the subject of this study, is a faithfulness-in-summarization question. A model can be poorly calibrated about its own beliefs yet faithful in relaying a reporter's, or the reverse, so the two properties must be measured separately. The distinction matters at the outset because the standard used to judge a summary here is fidelity to the source, not correspondence to the world.

That second question is the motivation for this study. Large language models now summarize safety text at scale, in triage pipelines that condense long reports into short abstracts, in search systems that index incident databases, and in analytic tools that cluster and count causes. Each of these uses substitutes a machine-written summary for the reporter's own words. If the summary preserves the reporter's epistemic marking, the substitution is harmless. If it does not, the record acquires a systematic distortion that no single reader is positioned to detect because detecting it would require comparing every summary to its source. A record in which possibilities have quietly become findings will mislead the very analyses that safety reporting is meant to inform.

The study addresses the preservation question directly, in two operational domains where the stakes of distortion are concrete, and asks three questions, with matched hypotheses stated in advance. First, when a model summarizes an incident report, does it change the reporter's epistemic marking, and in which direction? Hypothesis 1 is that models harden the reporter's uncertainty by default, presenting tentative claims as definite, and do so chiefly by deleting explicit hedges. Second, does an explicit instruction to preserve uncertainty repair the behavior? Hypothesis 2 is that it does not simply restore faithful marking but overcorrects, softening claims the reporter stated definitely, so that the obvious remedy introduces a second and opposite error. Third, is any default distortion merely an artifact of compression, given that a summary is shorter than its source and therefore drops words, some of which are hedges? Hypothesis 3 is that the effect is not reducible to length and survives a control for it. To keep the analysis honest, it was fixed in a study plan before any summary was generated, the outcome was measured two ways, each measurement instrument was validated against manual annotation, and a compression covariate was included so that a length explanation could be tested rather than assumed.

2 Related work

2.1 Epistemic marking in language and its detection

The linguistic machinery for expressing certainty is well-described. Hedges and boosters form a metadiscoursal system through which writers position themselves toward their claims, hedging to open space for doubt and boosting to close it Hyland, (2005). Epistemic modality, expressed through modal auxiliaries such as may, might, and must and through adverbs such as possibly and certainly, grades the speaker's commitment on a scale from bare possibility to confident inference Palmer, (2001). Evidentiality, the marking of the source of information, overlaps with this system because apparently and reportedly both weaken commitment and signal that the claim is secondhand. These devices are not interchangeable, and a summarizer that keeps modal verbs while dropping adverbial hedges will change the texture of a report even if it retains most of its propositional content. This study treats the categories separately for exactly that reason.

Computational treatment of these markers has a long precedent, and it also sets the terms for how they can be detected. The BioScope corpus annotated speculation and negation cues and their scope in biomedical text and showed that hedge cues are harder to detect reliably than negation because they are lexically diverse and context-dependent (Vincze et al., 2008). The CoNLL-2010 shared task then made hedge detection and scope resolution a benchmark problem and established that supervised, feature-based classifiers outperform fixed cue lists on held-out text, precisely because a cue such as may or appear is epistemic in some contexts and not others (Farkas et al., 2010). More recent work resolves that context dependence with transfer-learned neural models, for example, transformer classifiers for negation and speculation cues and their scope (Khandelwal and Sawant, 2020). A deterministic, lexicon-based detector, of the kind used here, inherits both the reproducibility of a fixed cue list and its central weakness, an inability to resolve polysemy; in Section 3.3, weakness on a hand-annotated sample is quantified, rather than asserting it, and the lexicon is situated as a transparent instrument to be triangulated against a model-based measure rather than as the last word on marker identity.

2.2 Large language models and the expression of uncertainty

A growing body of work asks whether language models express uncertainty faithfully, and the general finding is that they do not. A systematic study of faithful confidence calibration reported that models largely fail to communicate their uncertainty in words; standard prompting yields only marginal gains; some factuality-based calibration methods can even harm faithful expression (Liu et al., 2025). The downstream effect on readers is measurable: In a cross-language study, people overly relied on confident model phrasing, with a predicted overreliance risk near 10% for a strong model and approximately 40% for a weaker one, and the effect held across languages (Rathi et al., 2025). Larger models are reasonably calibrated on whether their own answers are correct when the question is posed in a suitable format (Kadavath et al., 2022), although eliciting that confidence in words is unreliable and sensitive to how the model is prompted (Xiong et al., 2024). A likely reason models sound more certain than they are is preference: Readers are biased against hedged text, so alignment with human feedback pushes models toward confident phrasing and away from doubt markers (Zhou et al., 2024). This bias is directly relevant here because it predicts both the default deletion of a reporter's hedges and the difficulty of restoring them without overshooting. All this literature, however, concerns the model as a source of claims, calibrating its own confidence. This study's question is the adjacent one introduced earlier: the model as a re-transmitter, where the standard is fidelity to the source.

2.3 Source-side preservation, faithfulness, and the gap

Preserving a source's uncertainty has been examined most directly in clinical text. A benchmark of diagnostic uncertainty found that models retained the original uncertainty cues poorly, often less than half the time, across a large annotated corpus (Du et al., 2026). This result establishes the phenomenon in one high-stakes domain. The behavior is a specific kind of unfaithfulness: Work on abstractive summarization has shown that models add content unsupported by the source and that faithfulness and factuality are distinct axes (Maynez et al., 2020), and a body of evaluation methods now targets that failure directly, from entailment-based inconsistency detectors (Laban et al., 2022) to question-answering consistency metrics (Wang et al., 2020), assessed within reusable human-annotated evaluation benchmarks (Fabbri et al., 2021). Dropping a reporter's hedge is a faithfulness failure of a particular epistemic kind that leaves the propositional content intact while changing its expressed certainty, and metrics built to catch fabricated or contradicted facts do not capture it because the fact is preserved and only its expressed certainty moves. Whether the phenomenon generalizes beyond clinical text to other high-consequence settings, and whether it is an artifact of compression, remains open. More broadly, a recent survey of pragmatics in the large-language-model era cataloged the data sets and evaluation methods now available and observed that discourse-level behavior and real-world, high-consequence settings are underrepresented in existing benchmarks (Ma et al., 2025). Aviation and maritime safety reporting are exactly such settings, and text mining these narratives is already an active area (Tanguy et al., 2016); however, its focus has been classification and retrieval rather than the model summaries' epistemic fidelity; to the author's knowledge, no prior study has measured whether models preserve reporters' expressed uncertainty there. This study fills that gap, adds a compression control not included in the clinical work, and validates both its measurement instruments against manual annotation.

3 Materials and methods

Table 1 defines the terms used throughout because several are closely related and are easy to conflate.

Table 1. Definitions of the key constructs, with a worked example of each.

The constructs are closely related, so a single passage makes the distinctions concrete. Consider a source sentence: “The gauge possibly misread and the crew may not have seen the vessel.” It carries two weakeners, the adverbial hedge possibly and the possibility modal may. A summary that reports “the gauge misread and the crew did not see the vessel” has hardened both claims: It deletes the two weakeners, raises the net commitment of the passage (a positive certainty shift), and drops the weakener retention for this passage to 0.0. A summary that instead reports “the gauge may have misread and it is unclear whether the crew saw the vessel” preserves the first hedge and, if the source had stated the second clause definitely, would have softened it. Epistemic fidelity is the property that both summaries are measured against: The first has low fidelity because it increased certainty the reporter did not express, and a faithful summary is one whose certainty shift is near zero. Hardening and softening are thus opposite failures of epistemic fidelity, and the passage-level certainty shift and the claim-level marker alignment (Section 3.4) are the two instruments that quantify them.

3.1 Data

Aviation sources were incident narratives from the ASRS, a confidential, voluntary reporting system whose de-identified records are a United States Government public-domain resource. Records were obtained through a public mirror of the ASRS database. Maritime sources were investigation-report summaries from the UK MAIB, published under the Open Government License v3.0 and retrieved through the official government content interface. The analysis set comprised 42 reports, 30 aviation and 12 maritime, whose source text carried at least one epistemic weakener because a report with no source uncertainty cannot exhibit preservation or distortion. Reports were drawn from the acquired public samples and retained under this criterion, with no further selection. Source narratives averaged roughly 230 words.

The two domains differ in register, and the difference is large enough to be measured rather than assumed. The ASRS narratives are first-person accounts written by the personnel involved, while the MAIB summaries are third-person investigator prose. Source weakener density is 1.94 markers per 100 words in the aviation narratives compared to 0.66 in the maritime summaries, a highly reliable difference (Welch t = 5.52, p < 0.0001). Because register is confounded with domain in this corpus, the domain comparison (Section 4.7) is treated as exploratory and is reanalyzed with source hedge density as a covariate so that a register explanation can be distinguished from a domain-specific model behavior. Because this confound cannot be removed within the present corpus, the domain result is reported only as evidence about register, not about aviation vs. maritime text as such, and its generalization is left to the larger, register-matched maritime sample described in Section 5.6. The report identifiers, retrieval URLs, and per-file SHA-256 checksums are listed in the Supplementary material, and one report is walked through end to end in Section 3.4 as a worked example.

3.2 Models and prompts

Six open-weight instruction-tuned models were evaluated through an OpenAI-compatible endpoint (Ollama Cloud): deepseek-v4-flash, glm-5.1, gpt-oss:120b, kimi-k2.6, minimax-m3, and qwen3.5. Decoding was greedy (temperature zero), with a fixed seed (0) and a ceiling of 1,024 new tokens, and each request logged the model identifier and a hash of the returned text so that every generation is traceable to its output.

Each report was summarized by each model under three prompts, giving 42 × 6 × 3 = 756 generations; after removing 106 empty outputs (see the following discussion), 650 summaries entered scoring. All models received the same system message, “You are a careful aviation and maritime safety analyst.” The plain condition (C1) instruction was “Summarize the following incident report in three to four sentences,” followed by the report text. The preserve condition (C2) issued the same request and appended an explicit instruction to keep the reporter's stance: “Preserve the reporter's expressed certainty and uncertainty exactly: keep hedges such as ‘possibly' or ‘appeared to,' and do not present a possibility as a definite fact.” The paraphrase condition (C2b) reworded that instruction to test whether any effect depended on its exact phrasing rather than its intent: “It is important that the summary reflects how sure or unsure the reporter was: if they wrote that something might have happened, do not state it as a definite fact.” The three prompts are reproduced verbatim in the Supplementary material.

Two features of this design bear on interpretation and are revisited in the Discussion. First, the preserve prompts are longer and more specific than the plain prompt, so a reduction in hardening under them could, in principle, reflect general instruction following rather than attention to epistemic marking specifically; the paraphrase condition C2b, which differs from C2 in length and wording but shares its intent, is a partial control for this, and a length-and-complexity-matched control prompt with unrelated content is identified as the definitive test (Section 5.6). The instruction lengths are 10 words (C1), 36 words (C2), and 40 words (C2b); because C2b is the longer of the two preserve prompts yet produces the smaller reduction in hardening (Section 4.5), instruction length alone cannot order the effect, which is evidence against a pure instruction-length account even before the definitive control is run. Second, empty model outputs, which occur when a model returns no visible answer, were recorded as missing and dropped before scoring rather than treated as summaries because an empty summary would otherwise register as a total loss of the source's hedges and inflate the apparent hardening. The 106 missing cases were concentrated in glm-5.1 and kimi-k2.6 under the preserve prompts, and in Section 4.6, a complete-case sensitivity analysis that excludes those two models is reported; in Section 4.8, a bounding analysis is added that imputes the missing outputs.

3.3 Epistemic-marker detection and its validation

Epistemic markers were detected in a lexicon of 143 entries built from the hedge and booster categories of (Hyland 2005) and the treatment of epistemic modality in (Palmer 2001). The lexicon groups markers into five categories, each carrying a direction. Hedges (57 entries) are weakeners that lower commitment, such as possibly, probably, apparently, appeared to, seems to, and suggests. Boosters (36 entries) are strengtheners, such as clearly, definitely, certainly, must have, and confirmed, that raise it. Epistemic modals (17 entries) are split by function: possibility modals, such as may, might, could, and may have are weakeners, while epistemic-necessity modals, such as must and will, are strengtheners. Evidentials (16 entries) mark the information source and are neutral in strength unless they also hedge, so according to and records show count as neutral, while apparently and reportedly count as weakeners. Approximators (17 entries) weaken numerical precision, such as about, approximately, roughly, and nearly. The complete category lists appear in the Supplementary material. Detection is deterministic, case-insensitive regular-expression matching on word boundaries, with multiword markers matched before their single-word substrings and overlapping matches resolved in favor of the longer marker.

Because a deterministic detector cannot resolve polysemy, its measurement error was quantified rather than merely acknowledged. A seeded sample of 45 source sentences (38 bearing markers and 7 without markers) was drawn, and each of the 46 tokens the detector flagged was judged in context as a true positive or a false positive, while each sentence was also scanned for genuine markers the detector missed. Against this hand annotation, the detector achieved a precision of 0.63, a recall of 0.91, and an F1 of 0.74. The errors were not evenly distributed across categories, and their distribution is what matters for the study's conclusions. Lexical hedges and boosters, the categories on which the headline result rests, were detected at a precision of 0.90; approximators, at 0.74; but epistemic modals, at only 0.23, because the fixed list cannot tell a deontic or ability should, could, or can from an epistemic one and surface matches such as the month May and the aviation term go around (which contains around) are counted as markers. Recall misses were almost entirely evidential perception and reporting verbs, such as advised, heard, and noticed, which the lexicon deliberately omits. Two consequences follow: Absolute marker counts, especially for the modal category, carry real measurement uncertainty, so they are reported as an instrument reading rather than a ground truth. But the direction of the study's effects does not depend on that error for three reasons: The same detector scores source and summary identically, so miscounts are symmetric and cancel in a difference; the headline finding is carried by the hedge category, where precision is high; and the low-precision modal category is the one that survives summarization best, so it works against, not for, the hardening result. A supervised or transformer-based detector (Farkas et al., 2010; Khandelwal and Sawant, 2020) would raise the modal-category precision, is a natural extension, and is committed to as the primary methodological improvement for the full-scale study (Section 5.6). The modal category's low precision is therefore a bounded, quantified limitation on absolute modal counts rather than a threat to the hedge-carried hardening result.

3.4 Outcome measures

Two outcomes were prespecified.

The passage-level outcome (O1) compares the source's and the summary's net epistemic commitment. Net commitment is the count of strengtheners minus the count of weakeners, normalized per 100 words, and the shift is the summary value minus the source value. A report counts as hardened when its source carried at least one weakener and the shift rises beyond a fixed threshold of +0.5 per 100 words and as softened when the shift falls below −0.5. Net commitment is computed over all five marker categories, so uncertainty preservation is measured from the full inventory of hedges, boosters, epistemic modals, evidentials, and approximators rather than from lexical hedges alone; the category-level breakdown in Section 4.7 shows how each category behaves. Two auxiliary measures accompany O1: weakener retention, the fraction of the distinct weakener markers in the source that still appear in the summary, computed overall and separately for each marker category, and the compression ratio, summary words divided by source words, carried into the analysis as a control for length.

The claim-level outcome (O2) aligns the source's and the summary's distinct epistemic weakeners and labels each: A source weakener absent from the summary is hardened (source tentative, summary definite), one that is retained is preserved, and a weakener that appears only in the summary is softened (source definite, summary tentative). The claim-level hardening rate is the count of hardened weakeners over all aligned weakeners; across the 650 scored summaries, the measure aligned 2,414 markers. This alignment, the measure used for all O2 results reported here, is deterministic and reproducible; a language-model judge intended to align claims semantically was also implemented, but as explained in Section 3.5, its archived run has been superseded by a corrected protocol whose validation is ongoing. The two measures are complementary: O1 aggregates net commitment over a whole passage, while O2 is anchored to individual markers, so agreement between them is a check on both.

A single report shows the measures at work, as rendered in Figure 1. In an ASRS narrative (report 1330846), a flight crew described an in-flight engine event in hedged terms, and the source carried 10 distinct weakeners, among them probably, it seems, seems to, could, assume, and relatively. Under the plain prompt (C1), gpt-oss:120b returned a summary that retained none of the source's distinct weakeners, added a single may, and stated the sequence as settled fact (“indicating an engine failure. The pilots declared the emergency”), a clear passage-level hardening. Under the preserve prompt (C2), the same model kept about, relatively, and it seems; reported the crew as “suspecting engine damage”; and hedged the reporter's causal claim (“it seems our operations program permits running the engines at higher temperatures, which may be causing our engines to fail”). The pair illustrates both effects the study measures at scale: the default deletion of hedges under plain summarization and their recovery, sometimes into over-recovery, under the preserve instruction.

3.5 Claim-level scoring, reliability, and validation

The claim-level outcome reported here is the deterministic marker alignment, defined in Section 3.4, in which a dropped source weakener counts as hardened, a retained one as preserved, and an added summary weakener as softened. A language-model judge was also run as an independent semantic check on the deterministic measure. In the original archived run, the judge models were given too small a token budget, returned no parseable output, and the pipeline fell back to the deterministic alignment for 98.9% of cases, so that run's inter-judge agreement compared the fallback with itself and is not used. The judge was therefore rerun with an adequate token budget and a single overall label per summary, hardened, softened, or preserved, which the models return reliably. Two judges, deepseek-v4-flash and minimax-m3, labeled the 650 summaries, returning a parseable label for 97.7% and 78.8% of cases, respectively; only parsed labels are used, and their agreement is reported in Section 4.1. The full judge prompt, the scoring code, and per-row provenance recording whether each label came from the model or the fallback are in the Supplementary material.

The claim-level measure was checked in three ways against references outside itself. First, it was compared with the passage-level O1 measure, which is computed from a different quantity (net commitment shift rather than marker set difference): The two agree on the direction of every condition effect and broadly on the rank order of models, with the same highest- and lowest-hardening model under the plain prompt and a Spearman rank correlation of 0.67. They agree less at the level of an individual report's dominant label, as expected for a passage-level and a marker-level instrument, which is why both are reported rather than one. Second, two independent language-model judges (Section 3.5) labeled each summary; they agreed with each other at Cohen's kappa = 0.55 (raw agreement 0.78) and with the deterministic measure at kappa = 0.26, a fair agreement that reflects the judges labeling most summaries preserved and being more conservative than the marker measure about counting a dropped hedge as hardening, and both independently reproduced the central pattern (Section 4.1). Third, the author manually adjudicated a seeded, condition-stratified sample of 24 source–summary pairs, reading each pair and assigning the dominant epistemic relationship without reference to the automated output. Agreement between this manual adjudication and the automated claim-level label was 0.79 (Cohen's kappa = 0.60), rising to 0.85 once ties are set aside, and more tellingly, the manual reading reproduced the central pattern on its own: Of eight plain-condition pairs, the author labeled five hardened and none softened, while of eight preserve-condition pairs, the author labeled all eight preserved or softer, none hardened. The instrument therefore captures the effect the study reports, corroborated by two independent language-model judges and a single human reader who is not one of the models; a larger multi-rater human study (Section 5.6) is the route to converting this reliability into full validity. For each summary, the distinct retained source weakeners were also recorded, overall and by marker category, which supports the category-level analysis in the results.

3.6 Statistical analysis

The observation unit is a single model summary of a single report, and multiple summaries are generated from each source report, so observations are not independent; an intrinsically easy- or hard-to-preserve report contributes correlated outcomes across models and conditions. The analysis addresses this non-independence directly by clustering on the source report. Hardening and softening rates were reported with 95% Wilson confidence intervals. A within-model McNemar exact test compared hardening and softening for each condition, and model-level comparisons used a Holm correction. The primary model of the hardening indicator (a report scored hardened at the passage level, as in O1) was a logistic regression of hardening on model, condition, and their interaction, with a compression-ratio covariate (summary words divided by source words) and cluster-robust standard errors with the report as the cluster, both accounting for the repeated use of each source and testing whether the condition effect survives a control for summary length. In model form, the specification was hardened ~ C(model) ^* C(condition) + compression_ratio, with a logit link and standard errors clustered by report over 38 report clusters. The preregistered primary specification was a mixed-effects logistic regression with random report intercepts; on the achieved sample, the fully saturated random-effects model (the model-by-condition interaction with a random report intercept) is not identifiable because the kimi-k2.6 preserve cell has n = 1, producing complete separation, so the cluster-robust specification is reported as the documented primary, and the mixed-effects models are reported for the estimable reduced specifications. To confirm that the condition effect is not an artifact of the fallback, in Section 4.8, this is refit as a mixed-effects (multilevel) logistic regression with a random intercept for source report, and separately with a random intercept for model, which directly address the nesting of summaries within reports and within models; the effect is unchanged. The modest number of clusters is a limitation of the validation-scale corpus (Section 5.5).

Effect sizes are reported as odds ratios with 95% confidence intervals, not as bare p-values, in Table 2 and are accompanied in Section 4.8 by predicted probabilities, a risk difference, and Cohen's h so that the practical size of the preserve effect can be read directly. A complete-case sensitivity analysis excluding the two models with high missingness (glm-5.1, kimi-k2.6) is also reported, together with a bounding analysis that imputes the missing outcomes under best- and worst-case assumptions (Section 4.8). Analyses were run in Python (statsmodels, scipy); the revision analyses that add the mixed-effects models, effect sizes, and missing-data bounds were implemented in numpy and scipy and validated to reproduce the primary cluster-robust estimates exactly.

Table 2. Primary logistic regression of the hardening indicator on model, condition, and a compression covariate, with cluster-robust standard errors by report (n = 589 model-report-condition observations, 38 report clusters).

Odds ratios with 95% confidence intervals; the plain condition and deepseek-v4-flash are the references. Model-by-condition interaction terms are omitted here for readability and reported in full in the Supplementary material; the complete-case refit on the four models without missingness gives the same preserve estimate (odds ratio 0.12, 95% CI [0.03, 0.49], p = 0.003).

3.7 Reproducibility

Raw data were kept immutable, all cleaning and analysis were scripted, seeds were fixed, and every model generation was cached with a content hash and logged. The archive contains the source corpora, the processed data, the per-claim alignment outputs, the two-judge language-model rerun outputs, and the judge protocol, the figures with their underlying data, per-source provenance with checksums, and a single command that regenerates the results. The Supplementary material collects the full marker lexicon, the verbatim summarization and judge prompts, the worked example, the model and decoding settings, the statistical specifications, the manual-validation protocols for the detector and the claim-level measure, an annotated subset of source–summary pairs for replicability (Section 5.6), and the data-provenance table with per-file SHA-256 checksums. A single command (reproduce.sh) regenerates the processed data, tables, and figures from the raw sources, and because every model generation is cached by content hash, the scoring and analysis reproduce exactly without re-querying any model. The archive and all code and data are openly available (see the Section Data availability statement).

4 Results

4.1 Reliability and validity of the claim-level measure

The claim-level marker-alignment measure was checked against references outside itself (Section 3.5). It agrees with the passage-level net-commitment measure on the direction of every condition effect and broadly on model rank order (same highest- and lowest-hardening model under the plain prompt, Spearman ρ = 0.67). Two independent language-model judges, rerun over all 650 summaries, add a semantic check: they agreed with each other (Cohen's kappa = 0.55; raw agreement 0.78) and the marker measure (kappa = 0.26), and each reproduced the central pattern on its own: The first judge labeled 90 summaries hardened compared to 12 softened under the plain prompt, reversing to 4 hardened and 28 softened under the preserve prompt, and the second showed the same reversal (85 compared to 8 under C1; 3 compared to 27 under C2). The judges label most summaries as preserved and are thus more conservative than the marker measure about absolute hardening, but they confirm its direction and the preserve-prompt reversal, so the effect holds across a marker-level, a passage-level, and a semantic instrument. The author's manual adjudication of 24 pairs agreed with the automated label at 0.79 (kappa = 0.60; 0.85 once ties are excluded) and reproduced the pattern on its own. The two-model agreement of 0.99 reported before this revision is not used because in that run both judges had fallen back to the deterministic alignment and the figure compared that fallback with itself (Section 3.5).

4.2 Detector precision and recall

The deterministic detector, audited against hand annotation of a 45-sentence sample, had a precision of 0.63 and a recall of 0.91. Precision was high for the lexical hedge and booster categories (0.90) that carry the main result, moderate for approximators (0.74), and low for epistemic modals (0.23), where polysemy between epistemic, deontic, and ability uses, together with surface collisions such as the month May, produces most false positives. Because the same instrument scored the source and summary, these errors are symmetric across the comparison, and the categories most affected are the ones that survive summarization best, so the measurement error does not manufacture the hardening effect; it is carried into the interpretation as a caveat on absolute marker counts (Section 5.5).

4.3 Default hardening under plain summarization

Under the plain prompt (C1), models presented tentative source claims as definite. Claim-level hardening ranged from 0.59 to 0.85 across the six models (Figure 2; glm-5.1 0.85, 95% CI [0.77, 0.90]; deepseek-v4-flash 0.76 [0.68, 0.82]; qwen3.5 0.73 [0.65, 0.79]; kimi-k2.6 0.68; gpt-oss:120b 0.64; minimax-m3 0.59). At the passage level, hardening exceeded softening for most models (Figure 3), and the difference was significant for glm-5.1 (rate difference +0.46, p = 0.0005) and qwen3.5 (+0.37, p = 0.007). The two measures agree on direction and rank models similarly, which is reassuring given that one aggregates net commitment over the whole passage and the other aligns individual markers. The loss fell most heavily on explicit hedges: The mean retention of source hedge words under C1 was about 0.08 (0.00–0.15 across models), while epistemic modal verbs survived far more often, at about 0.34 overall and 0.35 to 0.40 for every model but glm-5.1 (Figure 4). In other words, when a model shortens a report plainly, the adverbial hedges are the first casualties, and their removal is what turns a hedged observation into a flat assertion. Table 3 gives the passage-level rates and retention by model and condition.

Table 3. Passage-level hardening rate, softening rate, and mean weakener retention by model and condition, for reports whose source carried at least one weakener, with 95% Wilson intervals for the rates.

Effective n is 38 per model-condition cell, except glm-5.1 under C2 (n = 16) and C2b (n = 25), and kimi-k2.6 under C2 (n = 1), which is shown for completeness but omitted from interpretation.

4.4 Reversal under the preserve instruction

Asking models to preserve uncertainty carried them past preservation into softening. Claim-level hardening fell to a range of 0.24–0.40 under C2, and at the passage level, softening became the majority outcome, significantly so for deepseek-v4-flash (rate difference −0.50, p = 0.0002), qwen3.5 (−0.58, p < 0.001), and minimax-m3 (−0.37, p = 0.003). Mean weakener retention rose from roughly 0.14 to 0.38 under C1 to 0.59 to 0.66 under C2, which shows that the instruction changed behavior: The change overshot faithful preservation, inserting or retaining hedges that softened definite source claims. The models can therefore attend to epistemic marking when asked, and the failure under C1 is not an inability to represent uncertainty but a default disposition against carrying it across. The failure under C2 is a calibration problem: the same instruction that rescues a dropped hedge also softens a claim the reporter stated plainly.

4.5 Wording, compression, and the regression model

The paraphrase C2b reproduced the reduction in hardening at a smaller magnitude, so the effect does not depend on the exact phrasing of C2, though its size is sensitive to wording. Table 2 reports the primary logistic regression. The preserve condition had an odds ratio of 0.12 relative to the plain condition (95% CI [0.03, 0.49], p = 0.003); the paraphrase reduced hardening in the same direction but by less and not significantly (odds ratio 0.59, 95% CI [0.23, 1.55], p = 0.29). Model main effects and most model-by-condition interaction terms were not significant, consistent with a shared direction rather than a model-specific one.

Compression does not explain the condition effect, and a closer look shows why. The compression ratio was itself associated with passage-level hardening (odds ratio 2.93, 95% CI [1.22, 7.03], p = 0.016) but in the direction opposite to a simple “shorter drops words” account: Summaries that retained more of the source length were more, not less, likely to be scored as hardened. Decisively, the preserve conditions did not compress more aggressively than the plain condition; if anything, they were slightly longer (median compression ratio 0.59 under C2 and 0.50 under C2b compared to 0.46 under C1), so their sharply lower hardening cannot be attributed to shorter text, and the condition odds ratio holds with compression in the model. A category-level view makes the point sharper still. Under plain summarization, the retention of lexical hedges is near the floor (0.08) and essentially flat with respect to compression (Pearson r = 0.07, p = 0.44), so hedges are deleted whether the summary is long or short, whereas the epistemic modals' survival does depend on compression (r = 0.25, p = 0.003). The default loss of hedges is thus a targeted deletion, not a by-product of shortening, which is why the same-length summary can be more or less faithful, depending on the instruction.

4.6 Sensitivity to the high-missingness models

To check that the two models with high missingness did not drive the condition effect, the interaction logistic was refit on the four models with complete data (deepseek-v4-flash, gpt-oss:120b, minimax-m3, and qwen3.5). The preserve condition retained an odds ratio of 0.12 (95% CI [0.03, 0.49], p = 0.003), and the per-condition pattern was unchanged (plain hardening 0.41 compared to softening 0.24; preserve hardening 0.12 compared to softening 0.53), so the effect is not an artifact of the glm-5.1 and kimi-k2.6 attrition.

4.7 Marker categories and domain

The categories of epistemic marking were not lost uniformly, and this is where the distortion's shape is legible. The measure spans all five categories, so this breakdown also shows that uncertainty preservation is assessed from more than lexical hedges. Explicit hedge words, the adverbs and lexical hedges, such as possibly, appeared to, and apparently, were least likely to survive plain summarization, at about 0.08, while epistemic modal verbs, such as may and might, survived at about 0.34 and approximators at about 0.21. Under the preserve prompts, retention rose across categories, with hedges recovering the most in relative terms, from 0.08 to 0.54 (Figure 4). Evidential weakeners were too rare in the sources to estimate a retention rate (the hedging evidentials the lexicon marks, such as apparently and reportedly, appear only a handful of times), so the category is omitted from the retention figure; the reported measure of uncertainty preservation thus rests on the hedge, epistemic-modal, and approximator categories, which, together, account for essentially all source weakeners. The categories map onto functionally different kinds of uncertainty, which is useful for interpreting what is lost. Adverbial and lexical hedges most often carry the reporter's uncertainty about an inference or an observation (it appeared, probably); epistemic modals grade the possibility of a state or cause (may, could); approximators mark quantitative imprecision (about, roughly). On this reading, the default hardening falls hardest on inferential and observational hedging and least on graded possibility, so a plainly summarized report keeps its “maybes” while shedding the softer, more diffuse markers of a reporter thinking aloud. A finer decomposition of uncertainty by function, rather than by surface category, is a natural next step (Section 5.6).

Domain differences track this register difference rather than a domain-specific model behavior. Aviation narratives hardened more than maritime summaries under the plain prompt (hardening rate 0.47 vs. 0.28), and both domains softened under the preserve prompts. But the maritime sources are investigator summaries that carry far fewer hedges to begin with (Section 3.1); when source weakener density is entered as a covariate, the maritime indicator loses significance and even reverses sign (odds ratio 1.78, p = 0.35), while source weakener density is the significant predictor of hardening (odds ratio 3.38, 95% CI [1.67, 6.85], p = 0.0007). In other words, a report hardens in proportion to how much uncertainty it contained, and the apparent domain gap is a register effect: An investigator summary already spare in its hedging simply has less to lose. Two consequences follow for interpretation. First, because register and domain cannot be separated in this corpus, the result is evidence about how much a report's own hedging predicts its hardening, not about aviation vs. maritime as institutional domains, and it should not be read as a claim that models treat maritime text differently. Second, the maritime cell is small and register-confounded, so the domain finding does not by itself establish that the hardening behavior generalizes across registers; a register-matched maritime corpus, using full MAIB report findings sections rather than the already condensed investigator summaries, is the test of that generalization (Section 5.6). The maritime set is small (40–54 marker-bearing rows depending on condition), so this contrast is reported as exploratory.

4.8 Effect size, multilevel models, and missing-data sensitivity

The preserve effect size is large by conventional standards, not merely statistically detectable. In a pooled condition model with the compression covariate and cluster-robust standard errors, the preserve instruction lowers the predicted probability that a weakener-bearing report is hardened from 0.43 under the plain prompt to 0.13 (a risk difference of 0.30, a risk ratio of 3.3, and Cohen's h = 0.69, a medium-to-large effect), and the paraphrase lowers it to 0.23. At the claim level, the pooled hardening rate falls from 0.70 under the plain prompt to 0.35 under the preserve prompt, an absolute reduction of 35 percentage points and a halving in relative terms. In operational terms, where roughly one in two tentative source claims would be presented as definite under plain summarization, about one in eight would be under the preserve prompt, at the cost of the over-softening documented earlier, which is why neither prompt is a faithful target.

Because summaries are nested within source reports and within models, the condition effect was reestimated in mixed-effects (multilevel) logistic regressions that model this nesting explicitly (Table 4). A random intercept for source report, the preregistered structure, leaves the preserve odds ratio essentially unchanged at 0.13 (95% CI [0.08, 0.24]) and estimates a report-level intercept standard deviation of 1.38, an intraclass correlation of 0.37, confirming that reports differ substantially in how much they are hardened and that the cluster-robust primary analysis was addressing a real dependency. A random intercept for model instead leaves the effect at 0.20 (0.12–0.34) with negligible between-model variance (intercept SD 0.21, intraclass correlation [ICC] 0.01), consistent with the shared cross-model disposition reported throughout. The fully saturated preregistered model, which adds the model-by-condition interaction to the random report intercept, remains unidentifiable because the kimi-k2.6 preserve cell contains a single observation and produces complete separation; this is why the primary analysis uses the cluster-robust specification, which targets the same nesting through a robust variance estimator rather than an explicit random effect.

Table 4. Robustness of the preserve (C2) effect across model specifications.

Odds ratios (ORs) for the preserve condition relative to the plain condition, each with a compression covariate. The mixed-effects rows add a random intercept for the stated grouping and report its standard deviation and intraclass correlation (ICC).

Two questions about the 106 missing outputs bear on the result: whether the missingness is related to the epistemic content being measured and whether it could reverse the condition effect. Missingness was concentrated in glm-5.1 and kimi-k2.6 under the preserve prompts and predicted by source length (odds ratio 1.007 per source word, p < 0.001) but not by source weakener density (p = 0.66), so it is a decoding-and-model artifact, reasoning models exhausting their token budget on longer inputs, rather than a function of how hedged a source is; the outcome is therefore missing conditional on model and condition, not on the epistemic quantity of interest. A bounding analysis confirms the effect is robust to imputation (Table 5). Even under the worst case, in which every missing preserve output is counted as maximal hardening, the pooled preserve hardening rate (0.36) stays below the plain rate (0.43), and the preserve odds ratio stays below one (0.75), although attenuated and no longer significant; under complete-case and best-case imputation, the odds ratios are 0.20 and 0.15, respectively. The direction of the preserve effect thus does not depend on how the missing outputs are treated, while its exact magnitude, unsurprisingly, does.

Table 5. Missing data sensitivity.

Pooled passage-level hardening rate by condition (reports whose source carried a weakener) and the preserve odds ratio (OR), under complete-case analysis and under best- and worst-case imputation of the 106 missing outputs.

5 Discussion

5.1 A shared default among the models tested

The six open-weight models tested here share a default. When they summarize an incident report plainly, they harden the reporter's uncertainty; they do so chiefly by removing explicit hedges while retaining modal verbs. The direction matches what has been reported when models express their own confidence, where assertive language accompanies false claims and faithful calibration is hard to obtain (Liu et al., 2025; Rathi et al., 2025). This result carries the weak source preservation reported for clinical text (Du et al., 2026) into two safety-critical operational domains, showing the behavior is not idiosyncratic to one model or one prompt. The convergence across six independently trained models is consistent with the disposition being a property of how these instruction-tuned models compress text rather than a quirk of any one training pipeline, but the evidence is limited to six English-language models on aviation and maritime reports, so the claim is made about the models and settings tested, not about instruction-tuned models in general; establishing the latter would require a broader sample of models, languages, and domains.

5.2 The preserve instruction overshoots

An explicit instruction to preserve uncertainty does not restore faithful marking; it moves models the other way: keeping or adding hedges that soften definite source claims. Retention more than doubled under the instruction, showing the models can attend to epistemic marking when asked, so the C1 failure is one of disposition, and the C2 failure is one of calibration. The size of the swing tracked the exact wording, so a naive preserve prompt is not a reliable fix for faithful summarization. One caution is warranted in reading this result. The preserve prompts are longer and more specific than the plain prompt, so part of the change could reflect a general tendency to comply with a more elaborate instruction rather than attention to epistemic marking as such. Three observations bound that concern without dispelling it. First, the paraphrase C2b moves models in the same direction; because C2b (40 words) is longer than C2 (36 words) yet produces the smaller reduction in hardening, the effect ordering runs opposite to the instruction length ordering, which a pure instruction-length account cannot produce. Second, the change is specifically epistemic, concentrated on the retention and over-insertion of weakeners rather than on length or coverage. Third, the preserve summaries are not shorter than the plain ones, so the effect is not a compression side effect. A prompt matched to the preserve prompts in length and syntactic complexity but carrying an instruction unrelated to certainty would isolate the epistemic component cleanly, and it is the definitive control (Section 5.6). Even with that caution, the practical edge stands: the obvious remedy, telling the model to keep the hedges, introduces a second, opposite error, and a faithful summarizer needs a target that is neither the plain default nor the overcorrected preserve behavior.

5.3 Compression is not the whole story

Because a summary is shorter than its source, part of the hardening could in principle follow from length, but three results argue against a length account. The plain and preserve conditions compress to a similar degree, and the preserve conditions are, if anything, slightly longer, yet they differ sharply in hardening; the condition effect survives once compression is a covariate in the regression; and, at the category level, the deletion of hedges under plain summarization is flat with respect to compression, so hedges are dropped whether the summary is long or short. The compression covariate does carry its own signal but not the one a naive account predicts: The more the source length is retained is associated with more passage-level hardening, not less, which is consistent with longer summaries having more room to restate claims assertively, while the diffuse hedges are dropped regardless. The distinction matters for anyone tempted to treat the effect as an inevitable cost of shortening text. It is not because the same-length summary can be more or less faithful, depending on the instruction.

5.4 Implications for safety practice

Where a model summary stands in for a report in a searchable safety record, the two failure modes have different costs. A lost hedge can present a reporter's possibility as an apparent finding, which inflates the confidence of the record and can distort downstream cause analysis, for example, by overcounting a suspected factor as an established one. An overly eager preserve instruction can soften a firm observation into vagueness, which hides information the reporter meant to convey and can bury a clear signal under manufactured doubt. Both failures are invisible unless the summary is compared to its source, which is rarely done at scale. The stakes are institutional, as well as local: Aviation and maritime safety systems aggregate incident reports into the trend statistics and causal coding that inform regulatory attention, airworthiness directives, and safety recommendations, so a systematic hardening bias in machine summaries would bias which factors appear prominent in the aggregate record, precisely the kind of evidence that drives where regulators and operators direct scarce attention. Several concrete practices follow for anyone deploying models on safety text. First, treat epistemic fidelity as a first-class quality metric and report it alongside the familiar measures of coverage and factual consistency; the entailment- and question-answering-based consistency checks now standard for summarization (Wang et al., 2020; Laban et al., 2022) can be extended to score preserved, hardened, and softened claims rather than only contradicted ones, and the dual instrument used here, a transparent lexical measure triangulated against a model judge, is a workable template. Second, put a human in the loop where a hardened claim would carry weight, routing summaries whose measured certainty shift is large back for review rather than reviewing everything. Third, prefer a constrained generation target over a blanket instruction: a summarizer required to carry across the specific hedges detected in the source, evaluated against both the plain default and the overcorrecting preserve prompt, targets the loss where it actually occurs. Fourth, where models are trained or fine-tuned for this setting, add an objective that penalizes epistemic misrepresentation in both directions rather than only rewarding coverage. Fifth, where the record already feeds statistical analysis of causes, document the direction of the bias, toward overconfidence under the plain default, so that analysts can allow for it.

These practices suggest a concrete monitoring protocol for a deployed system, which need not wait for further research. A monitoring harness samples a fixed fraction of the summaries a system emits, computes for each the certainty shift and the weakener-retention and claim-alignment measures used here, and tracks their distribution over time on a control chart; a summary whose certainty shift exceeds a preset threshold, or a drift in the mean shift beyond control limits, raises a flag for human review or model reevaluation. Because the metrics are defined on the source–summary pair and require no gold labels, the harness runs continuously on production traffic, and a held-out set of human-adjudicated pairs calibrates the thresholds and audits the automated measures periodically. Such a harness turns epistemic fidelity from a one-off study result into an operational service-level indicator that can be specified in a procurement requirement or an assurance case.

The cost of these practices is modest and scales with the level of assurance sought. The lexical measure is essentially free: It is regular-expression matching that runs in milliseconds per summary on commodity CPU, so continuous monitoring of the certainty shift adds negligible overhead. A model-based claim judge adds one additional model call per summary audited; at current commodity inference prices, this is a fraction of a cent per summary and a second or two of latency, and because monitoring can sample rather than audit every output, its cost is a tunable fraction of generation cost rather than a doubling of it. The human-in-the-loop step is bounded by the flag rate: Routing only the summaries whose measured certainty shift is large—on these data, a minority of outputs—keeps review load far below reviewing everything, and the threshold trades reviewer time for missed distortions explicitly. The most expensive option, fine-tuning against an epistemic-fidelity objective, is a one-off training cost that a vendor can amortize across deployments. The practices therefore form a ladder from near-zero-cost monitoring to targeted human review to model retraining, and an operator can climb only as far as the consequence of distortion in their setting warrants.

5.5 Limitations

Several limitations qualify the claims. The claim-level outcome reported here is a deterministic marker alignment; the archived language-model judge had silently fallen back to that alignment, and the corrected two-model rerun, while reproducing the effect's direction, agrees with the marker measure only fairly (kappa 0.26) and labels most summaries preserved, so the absolute hardening rate should be read as a marker-level rather than a semantic quantity. The human check is a single author-annotator on 24 pairs; a larger, multi-rater human study with annotators drawn from outside the model set remains the clearest way to convert this reliability into validity. The deterministic detector has limited precision on the epistemic-modal category (0.23) because it cannot resolve deontic, ability, and epistemic uses or surface collisions such as the month May; the two measurement instruments therefore have distinct error profiles; although their errors are partly independent, so that agreement between them is more informative than either alone, a residual risk remains that a systematic bias shared by the lexicon and a model judge could inflate agreement, which the deterministic- vs.-model triangulation mitigates but does not eliminate. Two models, glm-5.1 and kimi-k2.6, returned many empty summaries under the preserve conditions, so their estimates there are underpowered and were read with caution; the complete-case analysis (Section 4.6) and the imputation bounds (Section 4.8) both show the condition effect does not depend on them. The corpus is a validation-scale set of 42 reports with 38 source clusters, which limits the number of clusters available to the regression and left the fully saturated preregistered mixed model unidentifiable; the reduced mixed-effects models (Section 4.8) do converge and reproduce the effect, but a larger corpus is needed to fit the full random-effects interaction. The maritime sample is small, and its sources are investigator summaries that are far less hedged than first-person aviation narratives, so the domain comparison is confounded with register and is reported as exploratory; a larger maritime corpus, and the use of full report findings sections in place of short summaries, would separate the two. The distinction between the preserve effect and general instruction-following is only partly controlled, as discussed earlier. Finally, the study measured fidelity to the source, not to the underlying event, and whether a hardened or softened summary misleads a human reader in a realistic decision task is a question for a controlled user study.

5.6 Future work

Several directions follow, and each maps to a limitation mentioned earlier. A human-annotated validation of the hardening measure, using multiple raters drawn from outside the evaluated model set and a shared annotation protocol of the kind established for summarization evaluation (Fabbri et al., 2021), would convert the present instrument-based claims into claims about how the summaries read to people. As a step toward this, an annotated subset of source–summary pairs, showing the detected markers and the per-pair hardening or softening label, is provided with the archive as a replication seed and a template for the multi-rater protocol, and the two-judge language-model rerun described in Section 3.5 provides a semantic cross-check that corroborates the direction of the effect; extending it to a larger panel of judges and to human raters drawn from outside the model set is the remaining validation step. A supervised or transformer-based epistemic-marker detector, trained on annotated data in the tradition of the CoNLL-2010 task and its neural successors (Farkas et al., 2010; Khandelwal and Sawant, 2020), would raise precision on the polysemous modal category and allow a context-sensitive reanalysis. On the intervention side, three prompting and training strategies invite direct comparison with both the plain default and the overcorrecting preserve prompt: a length-and-complexity-matched control prompt, with content unrelated to certainty, which would isolate the epistemic component of the preserve effect from general instruction-following—the prompt set and few-shot variants for this control prepared as a runnable extension of the generation pipeline; multi-turn strategies that first draft and then audit a summary against the source's markers; and fine-tuning or reinforcement-learning objectives that penalize epistemic misrepresentation in both directions. A larger, register-matched maritime corpus, built from full MAIB report findings sections rather than the already condensed investigator summaries, would separate register from domain and test whether the hardening behavior generalizes beyond the aviation-dominated sample here. A finer decomposition of uncertainty by function, separating uncertainty about observation, cause, and quantity rather than by surface marker category, would show whether models are especially prone to flattening particular kinds of epistemic content. Finally, connecting the text-level measures to a downstream task, such as an analyst judging cause from a summary vs. a source, would show whether the measured distortion changes decisions, which is the outcome that ultimately matters for safety reporting.

6 Conclusion

Across six open-weight models and two public safety-report corpora, plain summarization hardened reporters' expressed uncertainty, mainly by dropping explicit hedges, and an instruction to preserve uncertainty overshot into softening. The pattern held directionally across the models tested and two prompt wordings, survived a control for summary length, held in mixed-effects models and under adverse imputation of the missing outputs, and was reproduced by two automated measures, a passage-level lexical instrument and a claim-level marker alignment, that a human reader's adjudication of a subsample confirmed. The automated triage of safety reports should measure and report epistemic fidelity and should not rely on a single preserve instruction to keep the hedge.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/flang.2026.1953328/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Abeed Sarker, Emory University, United States

Reviewed by: Thangavel Murugan, United Arab Emirates University, United Arab Emirates

Chenchen Xie, China University of Geosciences, China