Evaluation of large language models for ICI-related liver injury with an emerging microbiome focus

1 Introduction

Immune checkpoint inhibitors (ICIs) restore antitumor immunity by blocking inhibitory pathways, including programmed cell death protein 1 (PD-1), programmed death ligand 1 (PD-L1), and cytotoxic T lymphocyte-associated antigen 4 (CTLA-4) (Ribas and Wolchok, 2018). This immune activation can disrupt self-tolerance and cause immune-related adverse events (irAEs) (Postow et al., 2018). ICI-related liver injury (ICI-LI) may require immunosuppression and interruption or permanent discontinuation of anticancer therapy (Brahmer et al., 2018; Dougan et al., 2021). Its clinical spectrum ranges from asymptomatic liver enzyme elevations to hepatitis, mixed injury, and cholangiopathy (Berry et al., 2023). Assessment therefore integrates clinical presentation, laboratory findings, imaging, treatment exposure, and competing causes.

The gut microbiome shapes systemic immunity and has been associated with both response and toxicity during ICI therapy (Gopalakrishnan et al., 2018; Matson et al., 2018; Regev et al., 2020; Andrews et al., 2021; Lee et al., 2022). Microbial composition and barrier integrity, together with microbiota-derived metabolites, can alter immune activity and host metabolism. The gut-liver axis links these processes to hepatic immune regulation: dysbiosis, barrier failure, and altered metabolites may disturb immune tolerance within the liver (Tripathi et al., 2018; Albillos et al., 2020). Yet microbiome findings vary across cancer types, treatment regimens, populations, sequencing methods, and analytical pipelines (Lee et al., 2022). Most microbiome evidence in this setting concerns ICI response or irAEs across cancer types rather than liver-specific toxicity. Reported signatures remain associations rather than validated biomarkers, and clinical interpretation must distinguish biological plausibility from evidence strong enough to guide care.

Large language model (LLM) chatbots are increasingly used by clinicians and patients to obtain biomedical information (Kung et al., 2023; Singhal et al., 2023). They can retrieve and summarize complex material within seconds. For evidence-intensive, cross-disciplinary problems such as ICI-LI, this speed may help users organize relevant knowledge. Their reliability, however, remains uncertain. Medical chatbot studies have largely examined general health education, question answering, or performance within a single specialty. Reported limitations include incomplete information, opaque sourcing, outdated knowledge, and technical language that can impair clinical interpretation and patient understanding (Ayers et al., 2023; Lee et al., 2023).

LLMs have rarely been tested on specialist questions that combine established clinical guidance with emerging microbiome evidence. The relevant task is clinical information synthesis: responses must organize evidence, state uncertainty, and avoid presenting associations as actionable biomarkers. Whether current platforms provide this calibrated synthesis remains unclear.

We therefore developed a disease-specific benchmark covering the broader clinical spectrum of ICI-LI with a focused microbiome component. We compared information quality, structure, transparency, and readability across platforms. The benchmark examines clinical information synthesis and evidence calibration, not diagnostic accuracy or clinical decision-making competence.

2 Methods

2.1 Study design and overview

This cross-sectional comparative study evaluated the quality, transparency, and readability of LLM-generated medical information about ICI-LI. We used a benchmark design with controlled inputs and a predefined collection procedure. Every question was submitted without modification, simplification, or added context. Because the study used only publicly available AI-generated text and involved no participants, patient data, biological specimens, or clinical interventions, ethics approval and informed consent were not required.

2.2 Development of the question dataset

We developed questions that sampled clinically relevant aspects of ICI-LI. Their content was based on current evidence for ICI-associated hepatotoxicity, microbiome-mediated immune regulation, clinical assessment, treatment, and translational research.

The final dataset comprised 25 questions (Table 1) across five domains: immunological mechanisms; the gut microbiome and immune regulation; clinical diagnosis and differential assessment; biomarkers and risk prediction; and therapeutic management and future applications. The benchmark therefore assessed ICI-LI broadly rather than treating the microbiome as an established clinical biomarker domain. Microbiome items were included to test how platforms handle a biologically plausible but clinically immature evidence base. All questions were specified before data collection and submitted unchanged to each chatbot.

Table 1. Benchmark questions on ICI-related liver injury with a focused microbiome component.

2.3 LLM selection and response collection

We evaluated four publicly available LLM-based chatbots: ChatGPT 5.5, Gemini 3.5 Flash, Perplexity, and Copilot. These platforms were selected because they represent widely accessible LLM-based systems with substantial user adoption and application in medical information retrieval. Each chatbot received the same questions without added prompts, context, or follow-up interactions. Responses were collected from the public web interfaces under default settings and retained as raw text. For readability analysis, we converted responses to plain text and removed symbols, bullet points, numbering, hyperlinks, citation markers, and separate reference lists without changing their wording or meaning. The evaluation was performed in July 2026 under publicly available default versions.

2.4 Assessment of chatbot-generated information

We assessed response quality and transparency with four established instruments: DISCERN, the Ensuring Quality Information for Patients (EQIP) tool, the Global Quality Score (GQS), and the JAMA Benchmark Criteria (Silberg et al., 1997; Charnock et al., 1999; Moult et al., 2004; Bernard et al., 2007). These instruments capture complementary features of treatment information, educational structure, perceived quality, and source transparency (Supplementary Table 2). Their scores were interpreted as indicators of information quality, structure, and transparency rather than direct measures of factual accuracy or clinical reasoning.

DISCERN assessed source transparency, completeness, and balance in treatment information (Charnock et al., 1999). Scores ranged from 16 to 80, with higher scores indicating better quality: excellent (63-80), good (51-62), fair (39-50), poor (27-38), or very poor (16-26).

The 20-item EQIP tool assessed the quality, structure, and usability of health education materials (Moult et al., 2004). Each Yes response received 1 point and each No response 0 points. We calculated the final score as (total score/20) × 100 and classified it as excellent (76-100), good (51-75), poor (26-50), or indicating extremely serious quality problems (0-25).

The GQS rated information organization, clarity, and usefulness on a five-point scale from 1 (very poor) to 5 (excellent) (Bernard et al., 2007). The JAMA Benchmark Criteria assessed authorship, attribution, disclosure, and currency; each criterion received 0 or 1 point, for a maximum score of 4 (Silberg et al., 1997).

Two medically trained reviewers independently rated every response (Ojha et al., 2021; Gkikas et al., 2022). Before formal assessment, they scored a random 10% sample to calibrate interpretation of the instruments. Reviewers were blinded to chatbot identity. Disagreements were discussed, and a third senior reviewer adjudicated unresolved cases.

2.5 Readability analysis

We measured linguistic complexity with six indices: the Automated Readability Index (ARI), Flesch Reading Ease Score (FRES), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index, and Simple Measure of Gobbledygook (SMOG) (Mac et al., 2022; Özduran and Hanci, 2022; Yilmaz Hanci, 2023; Hanci et al., 2024) (Supplementary Table 3). Higher grade-level scores indicate more difficult text, whereas higher FRES values indicate easier reading. A sixth-grade reading level was prespecified as the reference threshold for patient-oriented information, consistent with widely cited health-literacy guidance (Berkman et al., 2011; Ojha et al., 2021; Gkikas et al., 2022). The benchmark questions were primarily clinician-facing. Readability was analyzed separately to assess whether unmodified responses would also be suitable for patient-facing communication; it was not used as a measure of specialist content quality.

2.6 Statistical analysis

Continuous variables were summarized as median (interquartile range), and categorical variables as counts and percentages. Differences in quality and readability measures among chatbots were tested using the Kruskal-Wallis test. Significant omnibus results were followed by Dunn’s pairwise comparisons with Holm correction. Readability scores were compared with the prespecified sixth-grade threshold using the Wilcoxon signed-rank test. Kruskal-Wallis effect sizes were reported as epsilon squared (ϵ²). Analyses were performed using Python version 3.10.4.

3 Results

3.1 Characteristics and scope of the benchmark question dataset

The benchmark comprised 25 clinically oriented questions on LLM-generated information about ICI-LI. Five questions were assigned to each of five domains.

The domains covered immunological mechanisms, microbiome-mediated immune regulation, diagnosis and differential assessment, biomarkers and risk prediction, and therapeutic management with future translational applications. Together, they sampled mechanistic interpretation, evidence appraisal, clinical assessment, treatment decisions, and precision-medicine concepts. Supplementary Table 1 lists every question and its assigned domain.

3.2 Overall quality assessment reveals heterogeneous performance among LLM platforms

DISCERN, EQIP, and JAMA scores differed among the four chatbots, whereas GQS scores did not (Table 2). Kruskal-Wallis tests showed between-platform differences for DISCERN (P < 0.001, ϵ² = 0.212), EQIP (P < 0.001, ϵ² = 0.369), and JAMA (P < 0.001, ϵ² = 0.714). GQS scores were comparable (P = 0.826, ϵ² = 0.000).

Table 2. Quality scores of AI chatbots.

Values are shown as median (Q1, Q3). P values were obtained using the Kruskal-Wallis test; ϵ² denotes the Kruskal-Wallis effect size.

Copilot and Perplexity had higher median scores in selected quality domains (Figure 1). Their median DISCERN scores were both 43.00, compared with 38.00 for ChatGPT and Gemini. Median EQIP scores were 70.00 for Copilot and Perplexity, 55.00 for ChatGPT, and 50.00 for Gemini. Platform differences were therefore more apparent in completeness, reliability, and educational content than in perceived global quality.

Transparency produced the largest effect size. Median JAMA scores were 0 for ChatGPT and Gemini and 1 for Copilot and Perplexity (ϵ² = 0.714). By contrast, the identical median GQS scores across platforms suggest that global subjective assessments may fail to detect clinically meaningful differences identified by structured evaluation instruments.

3.3 Pairwise comparisons between LLM platforms

Dunn pairwise tests with Holm correction localized differences between platforms for each evaluation instrument (Table 3). Differences were concentrated in structured measures of information quality and transparency rather than the global impression score.

Table 3. Pairwise comparisons of AI chatbots (P values).

Cells show Holm-adjusted P values from Dunn’s post hoc test. NA indicates that no applicable comparison result was available.

For DISCERN, ChatGPT scored lower than Copilot and Perplexity; Gemini also scored lower than Copilot and Perplexity. The platforms thus differed in the reliability, completeness, and balance of treatment information.

EQIP showed the same grouping. ChatGPT differed from Copilot and Perplexity, as did Gemini. ChatGPT and Gemini did not differ, and neither did Copilot and Perplexity.

JAMA comparisons again separated ChatGPT and Gemini from Copilot and Perplexity. Differences were found for ChatGPT versus Copilot, ChatGPT versus Perplexity, Gemini versus Copilot, and Gemini versus Perplexity, but not within either pair. No GQS comparison was significant.

No platform was consistently superior across all instruments; performance depended on the dimension being measured.

3.4 Readability relative to a patient-facing benchmark

Responses from every platform exceeded the prespecified patient-facing reading level (Table 4 and Figure 2). Six indices assessed grade-level estimates and reading ease. Because the benchmark questions were primarily specialist-facing, this analysis addresses suitability for direct patient communication rather than the quality of clinician-facing communication.

Table 4. AI chatbot readability scores.

Values are shown as median (Q1, Q3). Sixth-grade thresholds were FRES >=80 and <6 for the grade-level indices.

All platforms exceeded the prespecified reading level for patient information. Median values ranged from 16.56 to 19.06 for ARI, 18.40 to 20.14 for GFI, 15.87 to 16.76 for FKGL, 18.39 to 21.29 for the Coleman-Liau Index, and 13.45 to 16.91 for SMOG. Each grade-level index was well above the sixth-grade threshold.

FRES values were also low, ranging from −4.37 to 11.31. Perplexity had the highest median FRES, but its responses remained far more difficult than the patient-oriented benchmark.

None of the four LLMs met the sixth-grade benchmark. Unmodified responses would therefore be difficult for many patients, although this finding does not assess their suitability for communication among clinicians.

3.5 Heterogeneous question-level performance patterns across LLM platforms

Question-level results varied within every platform (Figure 3). DISCERN, EQIP, GQS, and JAMA scores changed across benchmark items, showing that completeness, educational structure, and transparency depended on the question.

Between-platform contrasts were most visible in structured scores for questions requiring evidence appraisal or translation, including microbial toxicity signatures and evidentiary strength (Q8), barriers to biomarker translation (Q21), and integration of microbiome, clinical, and immunological data (Q24). In contrast, GQS ratings were largely similar across questions, including many items on core mechanisms and differential assessment.

Readability was more consistent: most responses to most questions exceeded recommended patient-education reading levels, despite variation across platforms. Translating specialized evidence into accessible language remained a shared weakness.

The observed variation was task- and metric-dependent and did not establish differences in clinical reasoning accuracy. Poor patient-facing readability persisted across the benchmark.

4 Discussion

This disease-specific benchmark demonstrates that current LLM-based chatbots exhibit heterogeneous performance on ICI-LI clinical information synthesis tasks with a focused microbiome component. Although all evaluated models generated structured medical responses, their performance varied across information quality, transparency, and readability relative to a patient-facing benchmark. Notably, different evaluation instruments captured distinct aspects of model performance, indicating that medical AI systems should not be assessed through a single global measure (Wu et al., 2025; Zhou et al., 2025). These findings support multidimensional evaluation of information quality and communication, but they do not establish clinical reasoning accuracy or biomedical competence.

ICI-LI represents a challenging setting for clinical AI evaluation because diagnosis and management require integration of complex biological and clinical information (Triantafyllou et al., 2025). Unlike conventional drug-induced liver injury, ICI-LI encompasses heterogeneous phenotypes, including hepatocellular injury, cholangitis-like manifestations, and immune-mediated inflammatory patterns, while requiring differentiation from competing causes of hepatic dysfunction (Haanen et al., 2022). Microbiome findings add a specific calibration problem because their biological plausibility exceeds their current clinical actionability (Gopalakrishnan et al., 2018). Differences in cancer populations, treatment regimens, sequencing strategies, and analytical approaches further complicate interpretation of microbiome findings (McCulloch et al., 2022). Most available microbiome data concern ICI response or irAEs across cancer types, whereas direct evidence for ICI-LI remains sparse. LLM responses should therefore distinguish biological plausibility, associative evidence, and clinically actionable evidence. In an emerging biomarker field, evidence calibration may be more clinically important than information volume.

The instruments differentiated platforms because they emphasize different response features. DISCERN and EQIP assess specified elements of content and structure, whereas JAMA assesses disclosure attributes. GQS provides a global impression and may compress differences in completeness, evidence attribution, and transparency. Its limited discrimination in this study supports complementary rather than isolated use of global ratings (Alderman et al., 2025; Blom and Pani, 2026). Higher JAMA scores may also reflect platform-level citation or source-display functions rather than superior biomedical competence. The JAMA Benchmark Criteria were originally developed for static online health resources and may not fully capture the characteristics of generative AI outputs (Agrawal et al., 2025). These instruments should therefore be interpreted as complementary indicators, and future AI-specific measures should assess factual accuracy, evidence traceability, and uncertainty communication directly.

The readability findings address a different task from specialist information synthesis. Complex terminology and high linguistic demands limit direct patient-facing use of unmodified responses (Ayers et al., 2023). Specialist responses may remain appropriate for clinicians despite high grade-level scores, whereas patient explanations of toxicity, monitoring, and treatment choices require deliberate adaptation. Patient-oriented systems should preserve medical accuracy while adjusting language complexity to health literacy (Raghu Subramanian et al., 2024; Mahajan and Powell, 2025).

Potential clinical uses should be defined by task risk. LLMs may assist with organizing literature, summarizing established guidance, and preparing clinician-reviewed explanations (Singhal et al., 2023). They should not independently interpret microbiome biomarkers for an individual patient, determine immunosuppressive treatment, or recommend ICI rechallenge. These higher-risk tasks require verified evidence and clinician judgment (van de Sande et al., 2024; Curcin et al., 2026). Future studies should test factual accuracy, calibration to evidence maturity, and the effect of LLM output on task-specific clinical decisions.

Several limitations should be acknowledged. First, only four publicly available platforms were evaluated at one time point, and subsequent updates may change performance. Second, the benchmark addressed one disease setting and included both general ICI-LI questions and a focused microbiome component, which limits generalizability. Third, the study assessed information quality, structure, transparency, and readability but not factual accuracy, diagnostic performance, or effects on clinical decisions. Fourth, DISCERN, EQIP, GQS, and the JAMA Benchmark Criteria were developed for consumer-oriented or web-based health information rather than generative responses to specialist biomedical questions. Their scores are complementary indicators and should not be interpreted as direct measures of medical competence. Finally, the sixth-grade threshold evaluates patient-facing accessibility; it does not define the appropriate complexity of clinician-facing communication. Studies involving clinicians and patients are needed to evaluate task-specific utility.

5 Conclusions

Current LLM chatbots differ in the quality, structure, and transparency of information generated for ICI-LI, and unmodified responses did not meet the patient-facing readability benchmark. The microbiome component showed the importance of calibrating claims to evidence maturity. Disease-specific multidimensional evaluation is needed before LLM-generated information can be considered reliable for clinical communication or decision support, particularly for biomarker interpretation and treatment decisions.

Generative AI statement

The author(s) declared that generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcimb.2026.1955356/full#supplementary-material

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Supplementary material Supplementary material at the source

Footnotes

Edited by: Ze Xiang, Zhejiang University, China

Reviewed by: Ye Zhang, the First Hospital of China Medical University, China

Rui Sun, Renmin Hospital of Wuhan University, China