Real-world performance of a Japanese browser-based hearing test: large-scale online usage data and concordance with clinical speech audiometry

1 Introduction

Hearing loss is a major public health concern. The World Health Organization (2021) estimates that more than 430 million people require rehabilitation for disabling hearing loss, a figure projected to exceed 700 million by 2050. Unaddressed hearing loss undermines communication and social participation and has been linked to loneliness, social isolation, and cognitive decline (World Health Organization, 2021; Shukla et al., 2020; Livingston et al., 2020). Adults commonly delay seeking care for several years after first noticing difficulty, with cost, stigma, access and limited audiology capacity repeatedly identified as barriers (Simpson et al., 2019). Conventional audiometry remains essential for diagnosis but, because it requires calibrated equipment in acoustically controlled rooms, is poorly suited to large-scale case finding or patient-driven monitoring between clinic visits (World Health Organization, 2021; Almufarrij et al., 2023).

Web- and app-based hearing tests have emerged as a low-cost, repeatable alternative for non-clinical settings, spanning pure-tone audiometry, speech-in-noise testing, word- or sentence-based recognition, and self-report questionnaires (Almufarrij et al., 2023; Wasmann et al., 2022). Pure-tone tasks are sensitive to ambient noise and hardware calibration, whereas speech-in-noise paradigms are comparatively robust in uncontrolled environments and may better reflect everyday listening ability (Almufarrij et al., 2023; Molander et al., 2013; Kwak et al., 2022; Smits and Houtgast, 2005; Potgieter et al., 2016). However, systematic reviews report wide variability in accuracy across platforms, and most validated tools have been developed in English-speaking populations and address a single modality at a time (Almufarrij et al., 2023; Mahomed et al., 2013; Saliba et al., 2017). This English-language dominance limits the generalisability of the existing evidence to the diverse global population of people with hearing loss, including the estimated 125 million speakers of Japanese, for whom culturally and linguistically appropriate digital tools are needed (World Health Organization, 2021). Comparatively little evidence is available for Japanese-language platforms or how online performance relates to the 67-S Japanese monosyllabic word recognition test used in routine clinical speech audiometry (Yamashita and Matsuhira, 2008).

Two properties of the standard Japanese materials are relevant at the outset. Monosyllabic word recognition in quiet is known to saturate in listeners with normal or near-normal hearing, and this ceiling has been documented even in cochlear-implant recipients, which motivated the development of implant-specific batteries, such as CI-2004 (Iba et al., 2011). Any platform built on these materials must therefore be expected to show a ceiling effect in the quiet condition, and the practical questions are not whether the ceiling exists but which alternative tasks avoid it and what measurement precision they achieve when self-administered without supervision.

Digital platforms are also frequently proposed for a second purpose: allowing patients undergoing aural rehabilitation to verify, between visits, whether a hearing-aid fitting or cochlear-implant mapping change has altered their day-to-day listening. Whether a self-administered test can support that use is a question of measurement error, not correlation, and it can only be answered if the reliability of repeated administration and the smallest change that exceeds measurement noise are known. Neither has been reported for a Japanese-language platform.

To address these gaps, we developed a Japanese-language browser-based hearing test platform that can be self-administered on a personal smartphone, tablet, or computer. The battery comprises a music perception task based on the Montreal Battery of Evaluation of Musical Abilities (MBEMA), speech perception in quiet, speech in noise at three signal-to-noise ratios (SNRs), and consonant identification, spanning basic music-related auditory abilities, phoneme-level identification, and word recognition under degraded listening. These modules measure behavioral performance on music- and speech-based listening tasks; they do not constitute an assessment of central auditory processing disorder, and we therefore refer throughout to auditory or listening performance rather than auditory processing.

The present study combines a large-scale observational analysis of real-world platform usage with a prospective paired comparison against conventional in-clinic Japanese speech audiometry. We aimed to (a) characterize score distributions across the major task types and quantify their upper-end clustering; (b) determine whether these distributions are an artifact of repeated testing by a minority of heavy users; (c) assess associations between music perception and speech-related performance; (d) estimate the reliability of repeated self-administration and the corresponding measurement error; and (e) quantify the agreement between the platform's 20-item speech check and the conventional 20-item 67-S monosyllabic word recognition test. This is a study of concordance and measurement properties. It was not designed to establish diagnostic accuracy: no pure-tone or otological reference standard was available, and no referral threshold, sensitivity, or specificity is reported.

2 Materials and methods

2.1 Study design

This retrospective observational analysis of real-world use of a Japanese-language browser-based hearing test platform between April 2023 and March 2026 was complemented by a prospective multicenter paired sub-study against conventional clinical speech audiometry conducted at participating Japanese otolaryngology departments.

2.2 Test platform and modules

The platform is a browser-based, self-administered system accessible on personal smartphones, tablets, and computers. It is openly available and does not require users to disclose demographic or clinical information beyond an optional statement about hearing-device use. Each attempt began with practice items and a device- and level-confirmation step, in which the participant played a calibration stimulus through their own hardware and adjusted the device's playback level to a comfortable listening level before the scored items began; practice and device-confirmation items were excluded from scoring. Because the platform was designed for unsupervised real-world use, transducer type (headphones, earphones, or the device's loudspeaker), absolute presentation level, and ambient noise were neither standardized nor recorded, and absolute acoustic calibration could not be guaranteed.

The full-length battery comprised four modules, all scored as per cent correct:

• Music perception (60 items). Based on the MBEMA, with 20 items each from three subtests: Melody (same/different judgement of two melodies differing in pitch contour), Rhythm (same/different judgement on temporal pattern), and Memory (yes/no recognition of whether a melody had been heard earlier in the session) (Peretz et al., 2013). Score granularity was 1.7 percentage points per item. An abbreviated 8-item version was also implemented.
• Speech perception in quiet (50 items). Japanese monosyllables drawn from the nationally recognized 57-S Japanese speech audiometry word list, presented without competing noise; participants selected the perceived syllable from on-screen closed-set response alternatives (Yamashita and Matsuhira, 2008). Score granularity was 2 percentage points per item. The abbreviated 10-item and 20-item versions, in which items were randomly sampled from the 57-S parent list at the start of each attempt, were implemented for brief screening (granularity 10 and 5 percentage points per item, respectively).
• Speech in noise (50 items at each of three SNRs). The same 57-S materials, with continuous background noise added at +10, +5, and 0 dB SNRs (Yamashita and Matsuhira, 2008). Score granularity was 2 percentage points per item.
• Consonant identification (14 items). Closed-set discrimination of 14 Japanese consonants (excluding palatalized sounds) in /aCa/ format, following the CI-2004 Japanese speech perception battery developed for cochlear-implant evaluation (Iba et al., 2011). Score granularity was 7.1 percentage points per item.

The choice of stimulus materials followed the platform's intended use. The 57-S and 67-S monosyllabic lists are the nationally standardized materials used in routine Japanese clinical speech audiometry and are therefore the only materials for which a comparable clinical measurement exists in ordinary practice; they are also appropriate for the general population users who constitute the large majority of the platform's user base. Because whole-word monosyllabic recognition is known to saturate, the battery deliberately also included tasks designed to avoid that ceiling: the consonant identification module, which is derived from the implant-oriented CI-2004 framework and lacks lexical redundancy (Iba et al., 2011), and the noise-loaded speech modules. No cochlear-implant-specific word list was implemented on the platform during the study period, so no such data were available for retrospective analysis.

2.3 Data extraction and data set structure

Response records were extracted from the platform's MySQL database and filtered by task identifier; practice and device-confirmation items were removed before score calculation. Two units of analysis were distinguished.

A test execution was any session in which a module was started. A completed attempt was an execution in which every scored item of the module received a response; incomplete executions have no module score by construction and could not be scored. The ratio of completed attempts to executions is reported as the completion rate for each module.

For the primary attempt-level distributions, repeated completed attempts by the same user were retained as separate observations because this component's aim was to characterize how the platform is actually used, in which repeated self-testing is a real and clinically relevant behavior. Because repeated attempts are not statistically independent and a small number of users contributed many attempts, every distribution was additionally recomputed with one completed attempt per user as a pre-specified sensitivity analysis (Table 1, Supplementary Figure S1).

Table 1. Score distributions of the six full-length modules for all completed attempts and one completed attempt per user.

Scores are percent correct. “One attempt per user” retains each user's most recent completed attempt, giving statistically independent observations. SNR, signal-to-noise ratio.

For all paired analyses, user-level data sets were constructed with one record per unique user per task pair, using each user's most recent completed attempt for each module concerned. This rule was verified computationally against the raw attempt-level records and reproduces all seven paired data sets exactly. Three families of paired data sets were constructed: (a) music perception vs. each speech-related task, for the between-task correlation analyses, (b) the short-form and standard-length versions of the same module (music perception: eight vs. 60 items; speech in quiet: 10 vs. 50 items) for the within-platform concordance analyses; and (c) the hospital and online speech-perception scores of the prospective sub-study. As a sensitivity analysis, all between-task correlations were recomputed using each user's first, rather than most recent, completed attempt (Supplementary Table S1).

2.4 Repeat-administration reliability and measurement error

Reliability of repeated self-administration was estimated for each full-length module from users who had completed the same module more than once. Attempts were ordered by the auto-incremented attempt identifier assigned at attempt creation, preserving the chronological order, and the first and second completed attempts by each such user were compared. The de-identified extract did not have attempt timestamps, so the elapsed interval between the two administrations is unknown and varied between users; these estimates therefore describe repeat-administration reliability under real-world conditions and are not equivalent to a formal test–retest study with a standardized interval.

For each module, we report the intraclass correlation coefficient with its confidence interval, the mean difference between the second and the first attempt (as an index of any practice effect), the 95% limits of agreement, the standard error of measurement (SEM, the square root of the residual mean square of the two-way analysis of variance), and the smallest detectable change at the 95% level (SDC95 = 1.96 × √2 × SEM), which is the smallest within-person change distinguishable from the measurement error (Weir, 2005; Terwee et al., 2007).

2.5 Sub-study comparing online and conventional in-clinic speech audiometry

A prospective multicentre paired sub-study was conducted at participating Japanese otolaryngology departments. Eligible patients attending for routine care underwent conventional in-clinic speech audiometry and were invited to complete the online speech-perception module unsupervised on their own device. Participants were instructed to use the same hearing-device configuration for both tests and were classified as aided if any hearing aid or cochlear implant was used in either ear and as unaided otherwise.

The hospital reference was the 20-item Japanese 67-S monosyllabic word recognition test, administered in a calibrated audiology booth by a licensed audiologist or trained audiometrician. Consistent with routine Japanese clinical practice, the test was presented at multiple intensities, and the maximum per cent-correct score across levels (Smax) was taken as the reference value (Yamashita and Matsuhira, 2008). The online measurement was the 20-item abbreviated version of the online speech-perception module, in which 20 items were randomly sampled from the 57-S parent list at the start of the attempt. Participants self-administered the measurement on their own smartphone, tablet, or computer at a playback level they set themselves.

The two measurements were obtained within days to a few weeks of each other, and this is the resolution of the available information: the exact interval, the location at which the participant chose to perform the online test, the transducer used and the ambient noise of that location were not recorded, and adherence to the instruction to keep the hearing-device configuration constant could not be verified. The online test was therefore performed exactly as an ordinary user would perform it, on hardware and in an environment of the participant's own choosing, and the comparison should be read as a comparison of two whole procedures rather than of two word lists.

This design was pragmatic by intention. The comparator was chosen to be the measurement that actually exists in Japanese clinical records so that the applied question—whether an unsupervised online score can be interpreted against what a clinician already has on file—could be addressed. Three consequences follow and are addressed in the Discussion. First, Smax is a best-performance measure obtained under optimized conditions, so a systematic negative bias of the self-administered score is a structural expectation, rather than an incidental finding, in this comparison. Second, because word list, transducer, presentation level, acoustic environment, and testing occasion all differed simultaneously, the study cannot attribute the disagreement to any one of them. Third, because days to weeks separated the two tests, genuine within-person variation over that period contributes to the observed differences in addition to measurement error.

2.6 Statistical analysis

Score distributions were summarized using the mean, median, standard deviation, interquartile range, range, skewness, and kurtosis. Upper-end clustering was indexed by the proportion of scores at or above 90%, specified a priori because it corresponds to no more than five errors on the 50-item modules and no more than two errors on the 20-item short form and is generally regarded as within the normal range for monosyllabic word recognition. Because any single threshold is to some extent arbitrary, the proportions at or above 95% and exactly 100% are reported alongside it for every module and every analysis set (Table 1). Normality was evaluated with the Shapiro–Wilk test on a random subsample of 500 observations per task (fixed seed) when the available n exceeded that threshold because Shapiro–Wilk rejects normality for trivial deviations at very large n; interpreting the distributional shape relied primarily on skewness, kurtosis, and visual inspection.

Between-task associations within the platform were quantified with Spearman rank-correlation coefficients, chosen because the score distributions were non-normal and the percentage outcomes were bounded, with frequent ties.

All three agreement analyses—repeat administration, short form vs. standard length, and online vs. hospital—used the same pre-specified set of complementary methods. The paired difference was tested against zero with the paired Student t-test and, as a non-parametric sensitivity analysis, the Wilcoxon signed-rank test. Linear association was quantified with the Pearson and Spearman correlation coefficients, with 95% confidence intervals for Pearson r obtained by Fisher's z-transformation. Absolute participant-level agreement was quantified with the two-way random-effects, single-rater, absolute-agreement intraclass correlation coefficient, ICC(2,1), interpreted according to the benchmarks of Koo and Li (2016) (< 0.50, poor; 0.50–0.75, moderate; 0.75–0.90, good; ≥0.90, excellent), with exact 95% confidence intervals derived from the F distribution as described by McGraw and Wong (1996). Bland–Altman analysis was used to characterize systematic bias and the spread of individual differences; 95% limits of agreement were computed as the mean difference ± 1.96 × SD of the difference, and confidence intervals for the bias and each limit of agreement were obtained by the method of (Bland and Altman 1986, 1999). Proportional bias and homoscedasticity were assessed by visual inspection of the Bland–Altman plots. To provide a directly interpretable index of agreement, we additionally report the proportion of participants whose two scores differed by no more than 10 and by no more than 20 percentage points.

The analyses were performed in R (version 4.6.1) and Python 3 (NumPy, pandas, SciPy). A two-sided p-value below 0.05 was considered significant. No adjustment for multiple comparisons was applied; the correlation and subgroup analyses are exploratory and are reported as such.

2.7 Ethical considerations

The study was conducted in accordance with the principles of the Declaration of Helsinki, and the protocol was approved by the Central Ethics Committee for Clinical Research, National Hospital Organization, Japan (approval number R6-0419001). For the retrospective platform-data component, the analysis used de-identified routine usage records, and the ethics committee waived the requirement for individual informed consent. For the prospective paired sub-study, electronic informed consent was obtained from every participant through the online platform before participation; the consent module described the purpose of the sub-study, the procedures, and the planned linkage of the online result with the corresponding in-clinic 67-S result, the right to withdraw without any effect on clinical care, and the de-identified handling of the linked data set. Only records with a valid recorded electronic consent were eligible for inclusion. For the single participant under 16 years of age, consent was additionally obtained from a parent or legal guardian.

3 Results

3.1 Platform use, module uptake, and completion

Between April 2023 and March 2026, the platform recorded 8,022 cumulative unique users and 86,959 test executions. The most frequently used modules were the brief screening versions: the 8-item music perception test (n = 36,068 executions, 41.5% of all executions), the 10-item speech-in-quiet test (n = 20,063, 23.1%), and the 10-item word-recognition test (n = 11,327, 13.0%). Among the six full-length modules analyzed here, executions totaled 4,333 for music perception, 3,190 for consonant identification, 1,163 for speech in quiet, and 643, 492, and 421 for speech in noise at +10, +5, and 0 dB SNRs, respectively.

Completed attempts and completion rates were 1,894/4,333 (43.7%) for music perception; 771/1,163 (66.3%) for speech in quiet; 458/643 (71.2%), 323/492 (65.7%), and 254/421 (60.3%) for speech in noise at +10, +5, and 0 dB SNR; and 2,738/3,190 (85.8%) for consonant identification. Completion was lowest for the longest module and highest for the shortest, which is consistent with test duration rather than task difficulty being the dominant determinant of abandonment. Because incomplete executions yield no module score, completers and non-completers cannot be compared on the outcome variable.

Across the six full-length modules, the median number of completed attempts per user was one (music perception: 1,738 users, mean 1.09, maximum 6; speech in quiet: 475 users, mean 1.62, maximum 110; speech in noise at +10/+5/0 dB SNRs: 339/259/156 users, mean 1.35/1.25/1.63, maximum 55/17/60; consonant identification: 1,412 users, mean 1.94, maximum 557), and only 6.5%−16.1% of users contributed more than one completed attempt. Activity was nevertheless concentrated in a minority for some modules: the five most active users contributed 1.5% of music perception attempts but 32.8% of consonant identification attempts and 31.5% of speech-in-noise attempts at 0 dB SNR.

Hearing-device information was available for 7,691 users. Of these, 7,289 (94.8%) were classified as unaided, and 402 (5.2%) reported using a hearing device: 247 indicated bilateral hearing aids (3.2%); 69, unilateral hearing aid (0.9%); 44, bilateral cochlear implants (0.6%); 27, unilateral cochlear implant (0.4%); and 15, bimodal (0.2%). No other demographic or clinical information was available for the retrospective component.

3.2 Score distributions

Descriptive statistics for all six full-length modules are given in Table 1, and the distributions are shown in Figure 1. All six attempt-level distributions were non-normal on Shapiro–Wilk testing (W ≤ 0.973, all ps < 0.001) and negatively skewed (skewness −0.60 to −1.59).

Music perception showed a broad upper-middle distribution (mean 80.7%, SD 9.9%, median 81.7%, range 31.7%−100%), with 20.8% of attempts at or above 90%, 5.3% at or above 95%, and 0.3% at 100% (Figure 1A). Speech perception in quiet showed pronounced upper-end clustering (mean 79.4%, SD 19.2%, median 86.0%), with 41.0% of attempts at or above 90%, 18.0% at or above 95%, and 2.3% at 100%, marked negative skewness (−1.59) and high kurtosis (5.57; Figure 1B). In the speech-in-noise tasks, mean accuracy declined progressively as the SNR worsened—72.0% at +10 dB, 63.3% at +5 dB, and 57.0% at 0 dB—and the upper-end clustering was progressively eliminated, with 17.9%, 0.9%, and 0% of attempts at or above 90%, respectively (Figures 1C–E). Consonant identification was intermediate (mean 76.8%, SD 20.7%, 35.3% ≥90%), although 17.5% of attempts were at exactly 100%, reflecting the 7.1-percentage-point score granularity of a 14-item test (Figure 1F).

3.3 Sensitivity of the distributions to repeated attempts

Because a minority of users contributed many attempts, all six distributions were recomputed using one completed attempt per user (Table 1, Supplementary Figure S1). This restriction removed the statistical dependence between observations and left the substantive findings unchanged or strengthened them.

For music perception and speech in noise at +10 and +5 dB SNRs, the user-level distributions were nearly identical to the attempt-level distributions (mean 80.9 vs. 80.7%, 74.7 vs. 72.0%, and 63.6 vs. 63.3%, respectively). For speech in quiet and for consonant identification, the user-level analysis produced higher scores and more upper-end clustering than the attempt-level analysis: the proportion at or above 90% rose from 41.0 to 57.9% for speech in quiet and from 35.3 to 51.1% for consonant identification. The repeated attempts of heavy users had therefore been diluting the ceiling effect rather than creating it. For speech in noise at 0 dB SNR, the user-level mean was lower (51.7 vs. 57.0%), likewise reinforcing the finding that the most adverse condition produces the broadest and lowest distribution with no upper-end clustering. The ordering of the six modules, the monotonic decline across SNRs, and the absence of any ceiling below +10 dB SNR held under both analyses and under all three ceiling thresholds.

3.4 Associations between music perception and speech-related performance

The user-level paired data sets comprised 294 users for music perception vs. speech in quiet, 202 vs. speech in noise at +10 dB, 156 at +5 dB, 81 at 0 dB and 453 vs. consonant identification. Music perception showed modest positive correlations with all speech-related tasks: speech in quiet (ρ = 0.352, p < 0.001; Figure 2A), speech in noise at +10 dB (ρ = 0.387, p < 0.001; Figure 2B), +5 dB (ρ = 0.367, p < 0.001; Figure 2C), and 0 dB (ρ = 0.352, p = 0.001; Figure 2D), and consonant identification (ρ = 0.305, p < 0.001; Figure 2E). Music perception therefore accounted for approximately 9%−15% of the variance related to speech outcomes. The correlations did not strengthen as the SNR worsened. Recomputing every correlation with each user's first rather than most recent completed attempt changed ρ by at most 0.017 and left every association significant (Supplementary Table S1).

3.5 Repeat-administration reliability and measurement error

Reliability of repeated self-administration was estimated in the 23–227 users per module who had completed the same module at least twice (Table 2, Figure 3). ICC(2,1) ranged from 0.58 (95% CI [0.44, 0.70]) for music perception to 0.85 (95% CI [0.65, 0.94]) for speech in noise at +5 dB SNR and was 0.83 (95% CI [0.73, 0.90]) for speech in quiet, 0.60 (95% CI [0.34, 0.78]) at +10 dB, 0.66 (95% CI [0.37, 0.84]) at 0 dB, and 0.65 (95% CI [0.57, 0.72]) for consonant identification.

Table 2. Repeat-administration reliability of the six full-length modules (first vs. second completed attempt of the same user).

Attempts were ordered by the attempt identifier assigned at attempt creation; the elapsed interval between the two administrations was not recorded and varied between users. CI, confidence interval; ICC, intraclass correlation coefficient; LoA, limits of agreement; pp, percentage points; SDC95, smallest detectable change at the 95% level (1.96 × √2 × SEM); SEM, standard error of measurement; SNR, signal-to-noise ratio.

On average, in every module, second attempts were slightly higher than first attempts, and the increase reached statistical significance for music perception (+2.9 percentage points, 95% CI [+1.2, +4.5], p = 0.001), consonant identification (+3.7, 95% CI [+1.4, +6.1], p = 0.002), and speech in noise at +5 dB SNR (+4.7, 95% CI [+0.5, +8.9], p = 0.029), consistent with a small practice effect on repeated self-administration.

The corresponding measurement error was substantial. The standard error of measurement ranged from 6.4 to 12.9 percentage points and the smallest detectable change from 17.6 percentage points (music perception) to 35.8 percentage points (consonant identification), with intermediate values for speech in quiet (26.5) and speech in noise at +10, +5, and 0 dB SNRs (27.0, 18.9, and 19.9, respectively); that is, a within-person change smaller than these values cannot be distinguished from measurement error under real-world self-administration.

3.6 Concordance between short-form and standard-length modules

To evaluate whether the brief screening versions could substitute for their full-length counterparts, user-level paired records were analyzed (Table 3, Figure 4).

Table 3. Agreement analyses with 95% confidence intervals.

pp, percentage points; LoA, limits of agreement; SEM, standard error of measurement; SDC95, smallest detectable change at the 95% level (1.96 × √2 × SEM); ICC, intraclass correlation coefficient. Subgroup analyses are exploratory.

For the music perception module (n = 312, eight vs. 60 items), the mean within-user difference (short–standard) was −1.7 percentage points (95% CI [−3.5, +0.04]; paired t = −1.93, p = 0.06; Wilcoxon p = 0.23). The two versions were positively but weakly associated (Pearson r = 0.38, 95% CI [0.29, 0.47]; Spearman ρ = 0.36), and agreement was poor, ICC(2,1) = 0.33 (95% CI [0.23, 0.43]), with 95% limits of agreement of −32.9 (95% CI [−35.9, −29.9]) to +29.4 (95% CI +26.4 to +32.5) percentage points and SDC95 = 31.2 percentage points. Only 48.4% of users scored within 10 percentage points of their standard-length score.

For the speech-in-quiet module (n = 71, 10 vs. 50 items), the mean difference was −2.9 percentage points (95% CI [−8.1, +2.3], p = 0.27), with a moderate association, Pearson r = 0.58 (95% CI [0.40, 0.72]), ICC(2,1) = 0.58 (95% CI [0.40, 0.72]) but 95% limits of agreement of −46.1 (95% CI [−55.1, −37.1]) to +40.3 (95% CI [+31.3, +49.3]) percentage points and SDC95 = 43.2 percentage points. Only 54.9% of users scored within 10 percentage points of their standard-length score.

3.7 Concordance with conventional in-clinic speech audiometry

The paired sub-study included 41 participants (25 unaided, 16 aided) who completed the in-clinic 67-S test and the unsupervised online check within days to a few weeks of each other; their characteristics are given in Table 4. Mean age was 50.9 years (SD 18.2; median 53, range 9–78), and 24 participants (58.5%) were women. Among participants, 16 used a hearing device: seven indicated a unilateral hearing aid; two bilateral hearing aids; three bilateral cochlear implants; one, a unilateral cochlear implant; and three, a bimodal configuration.

Table 4. Characteristics of the 41 participants in the prospective paired sub-study.

The in-clinic 67-S test and the unsupervised online test were performed within days to a few weeks of each other; participants were instructed to use the same hearing-device configuration for both. “Aided” denotes use of a hearing aid or cochlear implant in either ear. Smax, maximum per cent-correct score across presentation levels.

The group-level performance was high on both measures (hospital 67-S Smax: mean 88.4%, SD 17.0%, median 100%; online: mean 82.2%, SD 17.8%, median 90%) and was lower in aided than in unaided participants (unaided: hospital 94.4 ± 11.3%, online 88.8 ± 13.9%; aided: hospital 79.1 ± 20.3%, online 71.9 ± 18.8%). The reference test itself was close to the ceiling: 28 of 41 participants (68.3%) scored at or above 90%, and 21 (51.2%) scored exactly 100%, compared to 24 (58.5%) and eight (19.5%), respectively, on the online test.

The within-participant difference (online–hospital) had a mean of −6.2 percentage points (SD 14.5, 95% CI [−10.8, −1.6]), paired t(40) = −2.74, p = 0.009; Wilcoxon p = 0.011. The two methods were moderately associated (Pearson r = 0.65, 95% CI [0.43, 0.80], p < 0.001; Spearman ρ = 0.61, p < 0.001; Figure 5A) with moderate absolute agreement, ICC(2,1) = 0.62 (95% CI [0.37, 0.78]), a confidence interval that spans the poor-to-good range.

Bland–Altman analysis confirmed a bias of −6.2 percentage points with 95% limits of agreement of −34.7 (95% CI [−42.6, −26.8]) to +22.2 (95% CI [+14.4, +30.1]) percentage points (Figure 5B); visual inspection did not suggest proportional bias or marked heteroscedasticity. The corresponding SDC95 was 28.5 percentage points. Expressed directly, the online and hospital scores agreed within 10 percentage points (2 of 20 items) in 26 of 41 participants (63.4%) and within 20 percentage points in 36 of 41 (87.8%).

Mean differences were similar in the unaided and aided subgroups (−5.6 ± 14.8 vs. −7.2 ± 14.5 percentage points, p = 0.74) and were unrelated to age (Spearman ρ = 0.09, p = 0.56). Agreement, however, differed markedly between the subgroups: ICC(2,1) was 0.29 (95% CI [−0.07, 0.60]) in the unaided group, in which 76% of hospital scores were at or above 90% and the range of true performance was therefore severely restricted, but 0.69 (95% CI [0.31, 0.88]) in the aided group, in which performance was more widely distributed. These exploratory subgroup estimates are imprecise, but they indicate that the two tests' apparent agreement depends on the presence of participants with impaired word recognition and that the online test does not usefully discriminate within the normal range.

4 Discussion

This study characterized the real-world behavior and the measurement properties of a Japanese-language browser-based hearing test platform and compared its brief speech module with the conventional 67-S monosyllabic word recognition test performed in a calibrated booth. Four findings emerged: first, the platform's modules produced clearly different score distributions, and only the noise-loaded conditions avoided upper-end clustering. Second, this pattern was not an artifact of repeated testing by heavy users; restricting the analysis to one attempt per user preserved or accentuated it. Third, repeated self-administration of the same module yielded moderate-to-good reliability but the smallest detectable change of 18–36 percentage points. Fourth, the online speech check showed moderate group-level agreement with the clinical reference, with a small negative bias and limits of agreement spanning more than 55 percentage points.

Taken together, these results support a narrow, specific conclusion: the platform can be deployed at scale in a non-English-speaking population, produces interpretable and internally coherent score distributions, and places individuals in approximately the correct relative position on a clinical speech-audiometry scale. They do not support its use for individual clinical interpretation, and they do not address screening accuracy at all.

4.1 Which modules are informative under unsupervised conditions

The pronounced ceiling effect for speech in quiet was expected. The saturation of monosyllabic word recognition in listeners with normal or near-normal hearing is well established and is the reason cochlear-implant-specific batteries were developed in Japan (Iba et al., 2011; Almufarrij et al., 2023). Our contribution is to quantify the effect under genuinely unsupervised self-administration and show that it is robust to the threshold choice and the repeated-attempts treatment: between 41 and 58% of quiet-condition scores were at or above 90%, depending on the analysis set.

The practical consequence is that a quiet monosyllabic module is a poor primary outcome for a general population digital platform. The noise-loaded modules behaved as the speech-in-noise literature would predict, with mean accuracy falling monotonically from 72% at +10 dB to 57% at 0 dB SNR and the upper-end clustering disappearing entirely below +10 dB (Molander et al., 2013; Kwak et al., 2022; Smits and Houtgast, 2005; Potgieter et al., 2016). Consonant identification occupied an intermediate position, as expected for a closed-set phoneme task without lexical context; however, its 14-item length imposes a coarse 7.1-percentage-point granularity that limits its resolution and contributes to the 17.5% of attempts scoring exactly 100%. Based on these distributions, the +5 and 0 dB SNR conditions are the modules best suited to distinguishing between users of a self-administered platform.

The modest correlations between music perception and the speech-related tasks (ρ = 0.31–0.39, corresponding to approximately 9%−15% of shared variance) indicate that the MBEMA-based module measures something only weakly redundant with speech recognition. We do not attempt to identify the shared component here because the study measured none of the candidate mechanisms. The empirical implication is limited but useful: a music perception score cannot be substituted for a speech measure, and a multi-module composite might therefore add incremental information—a hypothesis our data can suggest but not test because the sub-study collected only the speech module for each participant.

4.2 What the reliability estimates mean for self-monitoring

Digital hearing platforms are frequently proposed as a way for patients to verify, between clinic visits, whether a hearing-aid fitting or an implant mapping change has altered their listening. The present data allow that proposal to be assessed quantitatively for the first time in a Japanese-language platform, and the answer is currently negative.

Repeat administration produced ICC(2,1) values of 0.58–0.85, which, by conventional benchmarks, are moderate to good (Koo and Li, 2016). However, reliability coefficients alone are misleading for a monitoring application because they depend on sample heterogeneity; the relevant quantity is the measurement error expressed on the score scale (Weir, 2005; Terwee et al., 2007). The smallest detectable change was 17.6 percentage points for the 60-item music module, 26.5 for the 50-item speech-in-quiet module, 18.9–27.0 for the speech-in-noise modules, and 35.8 for the 14-item consonant module. Changes smaller than these cannot be distinguished from noise in an individual user, and they are larger than the differences typically regarded as meaningful after a device adjustment. The same conclusion follows from the short-form analyses, where the smallest detectable change was 31.2 and 43.2 percentage points for the 8- and 10-item versions, respectively, and the sub-study, where it was 28.5 percentage points.

Longitudinal self-monitoring is therefore a potential application of this platform, not a demonstrated one. Making it viable would require reducing measurement error rather than reinterpreting the existing data, for example, by increasing the number of items, averaging repeated administrations, or standardizing the transducer. We also note a small but consistent practice effect (second attempts higher by 1.9–4.7 percentage points, significant in three of six modules), which would need to be accounted for in any longitudinal protocol.

4.3 Agreement with clinical speech audiometry

The online 20-item speech check and the clinical 67-S Smax were moderately associated, r = 0.65, ICC(2,1) = 0.62 (95% CI [0.37, 0.78]), indicating that individuals largely retained their relative position across two measurements that differed in word list, transducer, presentation level, acoustic environment and testing occasion. This result is meaningful for a test performed unsupervised, on unstandardised consumer hardware, in an environment that the participant chose and that was not recorded.

The negative bias of −6.2 percentage points is best understood as a structural consequence of the comparator rather than as a defect of the online test. Smax is a best-performance measure obtained at the optimal presentation level in a calibrated booth under examiner control; the online score was obtained at a level the participant chose on their own device. Comparing a self-administered score against a maximized reference should be expected to be negatively biased, and the observed magnitude—approximately one item in 20—is small relative to that expectation.

The limits of agreement are the more consequential finding. Spanning −34.7 to +22.2 percentage points, they mean that an individual's online score may be up to approximately 35 percentage points below or 22 percentage points above their clinical score, and the corresponding smallest detectable change of 28.5 percentage points exceeds any clinically meaningful difference in word recognition. Only 63% of participants agreed within two items. Four factors plausibly contribute: the 5-percentage-point granularity of a 20-item score so that a single misheard item moves the score appreciably; item-sampling variability because the 20 items were drawn at random from the parent list for each attempt; the absence of any acoustic standardization because participants chose the transducer and listening environment, which were not recorded; and the two tests being separated by days to weeks, which admits genuine within-person variation, as well as measurement error. These properties inflate variance without shifting central tendency, which is consistent with the Bland–Altman plot showing no proportional bias.

Some perspective on that magnitude is available from the reliability data. The smallest detectable change for the online–hospital comparison (28.5 percentage points) is of the same order as that obtained when the same user repeated the standard 50-item speech module on the platform (26.5 percentage points; Table 2). Much of the disagreement between the online check and the clinical test therefore appears to reflect the intrinsic instability of brief, unsupervised speech testing rather than a systematic difference between the two methods, which is why reducing measurement error, rather than recalibrating the online score against the clinical one, is the productive direction for development.

The subgroup analysis is instructive despite its imprecision. Absolute agreement was poor in the unaided group (ICC 0.29), in which three-quarters of clinical scores were at or above 90%, and moderate to good in the aided group (ICC 0.69), in which performance was widely distributed. Apparent agreement between the two measures therefore depends on the presence of participants with impaired word recognition; within the normal range, the online test does not resolve differences. This directly limits the plausibility of a screening application because case finding requires discrimination precisely at the boundary of the normal range.

4.4 What this study does not establish

Three claims a reader might expect from a digital hearing platform are not supported by these data and are not made.

First, this is not a diagnostic accuracy study. No pure-tone audiometry, otological examination, or clinical diagnosis of hearing loss was available as a reference standard; no referral threshold was defined; and neither sensitivity nor specificity was estimated. The data demonstrate concordance with clinical speech audiometry; they do not validate the platform for hearing-loss screening. Establishing screening performance would require a prospective study with a diagnostic reference standard, a pre-specified referral threshold, and a sample that spans the boundary of normal hearing.

Second, at the individual level, the platform is not a substitute for in-clinic speech audiometry for the reasons given in Section 4.3.

Third, the retrospective component describes platform behavior and score distributions, not a population's hearing status. The de-identified extract contained only the module identifier, an anonymized user identifier, and the item-level responses; age, sex, hearing status, reason for testing, playback hardware, and listening environment were unavailable, and because the platform is openly accessible, we cannot exclude that some accounts belong to clinicians or to people evaluating the tool rather than assessing their own hearing. Whether users who complete a module differ systematically from those who abandon one is also unknown; completion rates ranged from 43.7 to 85.8%, and an incomplete attempt has no score by construction, so the two groups cannot be compared on the outcome. If task difficulty drove abandonment, the reported distributions would be biased upward, inflating, rather than deflating, the ceiling effects reported here.

4.5 Strengths and limitations

The principal strengths of this study are the scale and the ecological validity of the platform data set, which allowed score distributions to be characterized across several listening conditions under unsupervised conditions difficult to reproduce in clinic-based research; the prospective sub-study, which used a nationally standardized reference test administered in a calibrated booth by trained examiners; and reporting measurement error alongside reliability, which converts qualitative claims about clinical utility into testable quantities.

In addition to the limitations mentioned in Section 4.4, several others constrain interpretation. Playback hardware and output level were neither recorded nor standardized, introducing uncontrolled acoustic variation; users listening through device loudspeakers would be expected to perform differently from those using close-coupled headphones, and this variance is subsumed within the wide limits of agreement. The ambient acoustic environments during self-administration could not be characterized. This applies to the sub-study and the platform data: the online test was self-administered unsupervised at a location, on a transducer, and at a level all chosen by the participant, of which none were recorded; the two measurements were separated by days to weeks rather than performed on one occasion; and instruction adherence to keep the hearing-device configuration constant could not be verified. The sub-study therefore compares two complete procedures and cannot attribute the disagreement between them to any single factor. Device-type classification (smartphone, tablet, computer) was available for only a subset of users, precluding analysis of platform-dependent differences. The reliability estimates were obtained from users who chose to repeat a module, whose motivations may differ from those of the general user, and the interval between administrations was not recorded; they therefore describe repeat administration under real-world conditions and are not a substitute for a formal test–retest study with a standardized interval. Attempt-level analyses retain repeated attempts and are descriptive rather than independent-sample inference, although the user-level sensitivity analysis showed that no conclusion depends on this choice. Participant motivation and sustained attention could not be monitored, which is particularly relevant for the longer modules. Finally, the sub-study was small (n = 41); recruited from otolaryngology clinic populations with consequent enrichment for hearing-device users (39 vs. 5.2% of the wider platform user base); pooled hearing-aid, cochlear-implant, and bimodal users into a single aided subgroup despite their different listening strategies; and included one participant under 16 years of age; its results should be regarded as exploratory.

4.6 Implications and future directions

This study provides real-world implementation evidence from a non-English, Japanese-language digital hearing assessment platform, addressing an imbalance in a literature dominated by English-language tools (Almufarrij et al., 2023; Mahomed et al., 2013; Saliba et al., 2017; Wasmann et al., 2022). The concrete implications for platform design are that noise-loaded modules should be preferred over quiet-condition monosyllabic tasks, that short forms of 8–10 items should be treated as engagement tools rather than measurements, and that measurement error, rather than correlation, should be the primary criterion when proposing a monitoring application.

Three studies follow directly: a diagnostic accuracy study with concurrent pure-tone audiometry and a pre-specified referral threshold is required before any screening claim can be made. A formal test–retest study with a standardized interval, a standardized transducer and a controlled environment is required to determine how much of the measurement error we observed is attributable to the uncontrolled listening conditions rather than to the tests themselves; a three-arm design comparing booth-calibrated, clinic-room self-administered, and home self-administered versions of the same online module would decompose these sources. Finally, a prospective study administering the full battery together with the clinical reference in the same participants would allow a multi-module composite to be evaluated, which the present sub-study could not do.

5 Conclusion

A self-administered, browser-based Japanese hearing test platform was used by more than 8,000 people over three years and produced score distributions that differentiated clearly between listening conditions, with pronounced upper-end clustering in quiet, progressive loss of that clustering as noise increased, and no dependence of these patterns on repeated testing by a minority of users. Its brief speech module showed moderate group-level concordance with conventional 67-S speech audiometry, with a small negative bias attributable to the maximized clinical reference and wide individual limits of agreement attributable to coarse score granularity, item sampling, and the absence of acoustic standardization. Repeat administration was moderately to well reproducible, but the smallest detectable change of 18–36 percentage points exceeds the changes that would need to be detected for individual monitoring. The platform is therefore best characterized at present as a scalable instrument for describing listening performance at the group level and engaging users with hearing self-assessment. Its use for hearing-loss screening and individual longitudinal monitoring remains an untested hypothesis that will require a diagnostic reference standard, a pre-specified referral threshold, and a reduction in measurement error.

Generative AI statement

The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI tools (ChatGPT and Claude) were used to improve the language, grammar, readability, and organization of the manuscript. All study design, data analysis, interpretation of the results, and final manuscript preparation were performed and verified by the author(s).

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fauot.2026.1921721/full#supplementary-material

Supplementary material Supplementary material at the source

Footnotes

Edited by: Chi Yhun Lo, Macquarie University, Australia

Reviewed by: Zahra Jeddi, Shiraz University of Medical Sciences, Iran

William Martens, Macquarie University, Australia