Early warning of academic risk from the first four weeks of online learning activity and assessment performance: a leakage-controlled, demographic-data-minimizing study on the OULAD benchmark

1 Introduction

Online and distance modes of higher education have expanded access for students who study alongside work and family commitments, yet they have long been associated with substantially higher dropout and failure rates than campus-based study (Simpson, 2013; Bawa, 2016). Attrition is costly for students, who accumulate debt and discouragement without credentials, and for institutions, whose funding and reputation increasingly depend on retention. Classical theories frame departure as a longitudinal process of weakening academic and social integration (Tinto, 1975). In distance settings, where informal face-to-face contact is scarce, retention research has repeatedly identified the early weeks of a course as the period in which disengagement typically begins (Simpson, 2013).

At the same time, the migration of teaching into virtual learning environments (VLEs) generates fine-grained digital traces of learning behavior—logins, clicks on materials, forum visits, quiz submissions—that institutions already collect as a by-product of instruction. Learning analytics and educational data mining have developed a substantial body of methods for turning such traces into predictions of academic outcomes (Romero and Ventura, 2020; Viberg et al., 2018; Pan et al., 2025). Within this agenda, early warning systems occupy a privileged place because prediction is only useful to teaching staff if it arrives in time to act (Ifenthaler and Yau, 2020).

Theoretically, the predictive value of VLE traces is grounded in two constructs. First, behavioral engagement—participation, effort, and persistence in learning activities—is a well-established antecedent of achievement (Fredricks et al., 2004), and log data offer scalable, if imperfect, behavioral indicators of it (Henrie et al., 2015; Bergdahl et al., 2024). Second, self-regulated learning theory holds that effective learners plan, monitor, and regulate their study (Zimmerman, 2002); in online environments such regulation should manifest as more regular engagement with course resources, and a systematic review links self-regulated learning strategies to achievement in online higher education (Broadbent and Poon, 2015). Empirically, indicators of regular study—active days, timely submission—tend to outpredict raw activity volume (You, 2016), and clickstream signals are often dominated by early assessment performance where the latter is available (Tempelaar et al., 2015; Conijn et al., 2017).

Section 2 reviews this literature in detail and identifies three gaps that motivate the present study. In brief: the discrimination achievable at a fixed, policy-relevant early time point is rarely the object of study; evaluation protocols on public educational benchmarks frequently permit leakage of information between training and test data, which makes reported accuracies difficult to compare; and predictive models commonly lean on background attributes whose collection sits uneasily with data-minimization principles, without the cost of forgoing them ever being quantified.

The objective of this study is to quantify, under a leakage-controlled and fully reproducible protocol, how much early warning is possible at the end of week four on the OULAD benchmark, and at what predictive-performance cost of data minimization. This cost is operationalized throughout as the change in predictive performance caused by removing all nine background/administrative features from the model while holding the sample, the split, the classifier, and every preprocessing step fixed; it is reported both as a difference in ROC-AUC and, because alert systems ration scarce tutoring capacity, as a difference in precision and recall among the highest-ranked students. Three research questions are addressed:

• RQ1: To what extent can final academic risk (fail or withdrawal) be predicted from VLE activity and early assessment performance observed during the first four weeks of a course?
• RQ2: How much incremental predictive value does early behavior add over background and administrative information available at enrollment—and what is the cost, in discrimination and ranking utility, of omitting that feature set entirely?
• RQ3: How well do models trained on past course presentations transfer to future presentations of the same modules, and to modules not seen during training?

The contribution of this work is deliberately not algorithmic. The dataset, the feature families, and the classifiers used here have all been studied before on OULAD (Wolff et al., 2013; Rizvi et al., 2019; Adnan et al., 2021; Borna et al., 2024), and no new model architecture is proposed. What the study contributes is evaluative and ethical: (i) a leakage-controlled protocol that directly measures how much cross-split student overlap record-level splitting would introduce on this benchmark, without claiming to estimate the resulting AUC bias; (ii) a head-to-head, uncertainty-quantified comparison of background and behavioral feature sets that prices the data-minimization choice in both discrimination and deployment terms; (iii) an assessment of generalization under a prospective year split and a descriptive unseen-module transport analysis, together with an empirical account of which module characteristics accompany good or poor transfer; and (iv) a fully traceable pipeline in which every reported number is written by the analysis code to labeled output files. Readers seeking higher headline accuracies will find them elsewhere in the OULAD literature; the argument here is that such figures are not comparable across studies until the evaluation protocol is pinned down, and that pinning it down is a precondition for the field’s cumulative progress.

2 Related work

2.1 Predicting academic risk in online and distance higher education

Pioneering early warning systems demonstrated both the promise and the practical difficulties of the idea. Course Signals at Purdue combined grades, effort, and background characteristics into a traffic-light indicator associated with improved retention (Arnold and Pistilli, 2012); Macfadyen and Dawson showed that simple counts of LMS activity could explain a substantial share of variance in final grades and flag failing students (Macfadyen and Dawson, 2010); the Open Academic Analytics Initiative ported early-alert models across institutions (Jayaprakash et al., 2014); and dashboard-based alerts have since been embedded in institutional tutoring workflows (Foster and Siddle, 2020) and adopted at scale, though unevenly, across the sector (Márquez et al., 2024). Recent syntheses map a large and methodologically heterogeneous body of prediction studies (Pan et al., 2025; Lopez-Muñoz et al., 2026), while reviews continue to caution that rigorous evidence connecting analytics to improved study success remains thin (Ifenthaler and Yau, 2020; Viberg et al., 2018).

Two strands of this literature bear directly on the present study. The first concerns how early useful prediction becomes possible. Ensemble models compared at successive fractions of course length show accuracy rising monotonically with the observation window (Adnan et al., 2021), which makes any single reported accuracy uninterpretable without its time point; work using deliberately limited early-course data reports correspondingly more modest performance (Azizah et al., 2024). The second concerns what the traces carry: sequential models of learning trajectories (Gupta et al., 2022), textual records (Phan et al., 2023), and clustering of digital traces (Pecuchova and Drlik, 2024) all improve on raw activity counts, but multi-course evidence indicates that early grades typically dominate clickstream volume where both are available (Tempelaar et al., 2015; Conijn et al., 2017).

2.2 OULAD as a benchmark

The Open University Learning Analytics Dataset (OULAD; Kuzilek et al., 2017) has become a standard public benchmark for this research line, accompanying the Open University’s own production systems such as OU Analyse (Herodotou et al., 2019). Early work at the Open University predicted failure from VLE clicking behavior combined with assessment scores (Wolff et al., 2013), and addressed the cold-start problem of new course presentations (Hlosta et al., 2017). On OULAD itself, deep learning models have been applied to clickstream features (Waheed et al., 2020; Fazil et al., 2024), ensemble models have been compared at successive fractions of course length (Adnan et al., 2021), demographic effects have been analyzed with decision trees (Rizvi et al., 2019), and work in this journal compared random forests, gradient boosting, and recurrent networks on click data, reporting strong performance for predicting withdrawal but difficulty separating adjacent outcome categories (Borna et al., 2024). A recent systematic review confirms the breadth of predictive-modeling approaches evaluated on OULAD and the continuing need for comparable protocols (Jin et al., 2024). The breadth of this activity is what makes the benchmark valuable; it is also what makes the comparability of results across studies a pressing methodological question.

2.3 Evaluation protocols, leakage, and comparability

Students frequently enroll in several modules or presentations, so record-level random splits allow the same student to appear in both training and test data. Model selection performed on the test set, and imputation or scaling fitted on pooled data, add further optimism. Such leakage is now recognized as a principal driver of the reproducibility crisis in applied machine learning (Kapoor and Narayanan, 2023), and a recent survey catalogues the specific mechanisms by which it enters conventional pipelines (Apicella et al., 2025). In education, both reviews and multi-course studies echo the concern that incomparable or overoptimistic setups hinder cumulative progress (Gardner and Brooks, 2018; Gašević et al., 2016). The practical consequence is that a headline AUC on OULAD conveys little unless the splitting unit, the selection procedure, the imputation scope, and the observation window are all stated; the present study fixes all four and reports what each choice is worth.

2.4 Privacy, data minimization, and fairness

Predictive models commonly lean on demographic attributes—age, neighborhood deprivation, prior education—that are ethically delicate. Profiling students by who they are rather than what they do raises fairness concerns (Baker and Hawn, 2022; Opoku et al., 2025), sits uneasily with data-minimization principles in privacy frameworks (Drachsler and Greller, 2016; Pardo and Siemens, 2014), and can entrench deficit narratives about disadvantaged groups (Slade and Prinsloo, 2013). Empirical work documents that students themselves distinguish sharply between analytics uses they regard as pedagogically legitimate and those they experience as surveillance (Soffer and Cohen, 2024), that institutional practice around student privacy is often underdeveloped (Francis et al., 2023), and that privacy is neither a universal nor a purely individual construct across educational contexts (Prinsloo and Kaliisa, 2025). In parallel, a fairness literature has shown that accuracy and equity trade off in student-performance models (Opoku et al., 2025), that fairness risks can enter at measurement, model, and deployment stages (Kizilcec and Lee, 2022), that learning analytics for disabled and disadvantaged students remains an underdeveloped area (Khalil et al., 2024), that explainability and performance can be balanced in dropout prediction (Zanellati et al., 2024), and that early warning systems evaluated only on aggregate accuracy may conceal substantial subgroup disparities (El Jihaoui et al., 2026; Xia and Chen, 2026).

Crucially for the present study, privacy and fairness are distinct objectives that are easily conflated. Removing protected attributes from the input space is a data-minimization measure; it does not, by itself, remove disparate performance, because behavioral features can act as proxies for group membership. Whether a deployable early warning model can forgo demographic inputs with acceptable loss of accuracy—and what happens to subgroup disparities when it does—is therefore a question of practical ethics as much as of machine learning, yet demographic and behavioral feature sets are seldom compared head-to-head under a rigorous protocol (but see Rizvi et al., 2019, for an analysis of demographic effects on OULAD).

2.5 Gaps addressed by this study

Three gaps follow. First, the discrimination achievable at a fixed, policy-relevant early time point—such as the end of week four, when tutoring resources can still be redirected—is rarely the focus, and is rarely reported with uncertainty. Second, evaluation protocols on OULAD are seldom leakage-controlled, and the magnitude of the resulting optimism has not been quantified on the benchmark itself. Third, the cost of omitting the background/administrative feature set has not been quantified, either in discrimination or in the ranking terms that govern how alerts are actually used, and the consequences of such omission for subgroup disparities have not been tested. The present study addresses all three.

3 Materials and methods

3.1 Dataset and provenance

OULAD (Kuzilek et al., 2017) contains anonymized records from seven modules and 22 module presentations (2013–2014) of the Open University, the largest distance-teaching university in the United Kingdom. The dataset covers 32,593 course enrollments by 28,785 distinct students, together with 10,655,280 rows of VLE interaction logs totaling 39,605,099 clicks, spanning from 25 days before to 269 days after module start. Final results are distributed as Pass 37.9%, Withdrawn 31.2%, Fail 21.6%, and Distinction 9.3%. The binary target at-risk was defined as Fail or Withdrawn (52.8% of all enrollments). OULAD is published under a CC-BY 4.0 license. Here, an enrollment denotes one student–module–presentation record, and a presentation is a dated offering of an anonymized module.

This study used the copy mirrored in the UCI Machine Learning Repository (dataset #349). Because a mirrored copy need not be identical to the original release, every table was checked against the counts published in the original data descriptor (Kuzilek et al., 2017). All seven data tables matched exactly: 32,593 student–module records, 32,593 registration entries, 22 presentations across 7 modules, 10,655,280 VLE log entries, 6,364 VLE materials, 206 assessments, and 173,912 assessment entries. One discrepancy was found, and it lies in documentation rather than data: the plain-text OULAD.names file distributed with the UCI mirror reports 32,953 records for both the student–module and registration tables. This number appears to be a digit transposition of 32,593: the accompanying studentInfo.csv and studentRegistration.csv files each contain 32,593 records, as reported in the data descriptor. No modification of the data tables themselves was detected. Byte counts and MD5 checksums for all seven CSV data tables used in the analysis are listed in the Supplementary Material so that readers can confirm they are working with the same copy. Beyond the single exclusion rule described next, no filtering, deduplication, outlier removal, class balancing, or activity-based screening was applied.

3.2 Prediction scenario and sample construction

The prediction time point was fixed at day 28 after module start, corresponding to the end of the fourth teaching week—early enough for tutoring resources to be redirected, yet late enough for behavioral signal to accumulate. To avoid outcome leakage, all enrollments whose official unregistration date preceded day 28 were excluded, because their outcome was already determined at prediction time. Table 1 reports the resulting sample flow and reconciles it with the published dataset, by module. The exclusion removed 5,040 of 32,593 records (32,593−5,040=27,553). Of those excluded, 5,032 ended as Withdrawn and 8 as Fail, and their median unregistration date was day −2 (interquartile range −25 to 12), confirming that the rule removes records whose fate was sealed at or before the prediction point rather than an arbitrary slice of the cohort. The retained analytic sample comprises 27,553 enrollments with an at-risk rate of 44.2%, composed of 12,361 Pass, 7,044 Fail, 5,124 Withdrawn, and 3,024 Distinction records. Accordingly, the estimand is risk among enrollments still active at day 28, not risk among all students initially registered; “withdrawal” below means subsequent withdrawal after that prediction point.

Table 1. Sample flow and reconciliation with the published dataset.

Counts in the “Data descriptor” column are those reported in Table 1 of Kuzilek et al. (2017); the downloaded UCI copy matched them exactly for every module. The single exclusion rule removes enrollments formally unregistered before the day-28 prediction point, whose outcome was already determined.

All features use only information available before day 28. Eighteen retained enrollments have registration dates on or after day 28 (latest: day 167) and would not yet be observable in a strict day-28 deployment; a sensitivity analysis excluding them is reported in Section 4.9.

3.3 Features

Table 2 defines the two feature sets. Thirteen behavioral features were computed from VLE logs over days −25 to 27 (i.e., up to day 28, including the pre-course warm-up period): cumulative clicks, active days, distinct resources and activity types, weekly click counts, an engagement-trend contrast (weeks 3–4 minus weeks 1–2), early assessment submissions and mean score, and a zero-activity indicator. In the retained sample, 3.4% of enrollments had no VLE activity at all, and the median cumulative click count was 221. The comparison set contains nine background and administrative features available from institutional records: six demographic or socioeconomic attributes (gender, region, highest prior education, IMD band, age band, and declared disability) and three prior-study or registration variables (previous attempts, credits studied, and registration date). We call this the background/administrative set; it should not be read as nine strictly demographic attributes.

Table 2. Feature definitions.

Behavioral features use VLE and assessment records dated before day 28 (including the pre-course period, days −25 to −1); background/administrative features come from institutional records.

OULAD records the submission date and eventual score but not when marking was completed or the score was released. The primary analysis therefore represents the common deployment condition in which a score from a pre-day-28 submission is available by the prediction point. A strict availability sensitivity removes early_mean_score entirely while retaining submission counts and all VLE features (Section 4.9). Early assessment scores also include results “banked” from a previous attempt at the module (7.5% of early assessment records, affecting 647 enrollments); these are known at prediction time but do not reflect performance in the current presentation, so a sensitivity analysis excluding them is reported in Section 4.9. Behavioral count features for students without VLE or assessment records were set to zero, which represents true zero engagement rather than missingness. Missing values of early mean score and registration date were retained as missing and imputed inside the modeling pipeline using training-fold medians with added missing-indicator columns. This avoids imputation leakage and the distortion that sentinel values impose on linear models.

3.4 Models and evaluation protocol

Three classifiers were compared: ℓ2-regularized logistic regression, random forest (Breiman, 2001), and histogram-based gradient boosting (HistGB), all implemented in scikit-learn (Pedregosa et al., 2011). Each was crossed with three feature sets (background/administrative only, behavior only, all features), yielding a full 3×3 ablation. Categorical features were one-hot encoded; numeric features were median-imputed (with indicators) and, for logistic regression, standardized; all preprocessing was fitted within training folds only. The nine candidate pipelines were fixed in advance: logistic regression used C=1 and a 2,000-iteration limit, random forest used 300 trees and a minimum leaf size of 5, and HistGB used scikit-learn defaults; no hyperparameter search was conducted.

Grouped splitting. Enrollments were split 80/20 into training and test sets with grouping by student identifier (GroupShuffleSplit, seed 42; ntrain=22,035, ntest=5,518; at-risk rates 44.2% and 44.1%), guaranteeing that no student contributes records to both sets. The test set contains 4,970 unique students; 527 contribute more than one enrollment. As a control, the cross-split overlap that record-level random splitting would have introduced was quantified: 852 students would have appeared in both sets, affecting 15.7% of test records.

Model selection. The in-pool configuration was selected from the nine fixed candidates using mean ROC-AUC in five-fold student-grouped cross-validation (GroupKFold) on the training set; the test set was used once per candidate for final reporting.

Uncertainty and model comparison. Main AUC, outcome, and subgroup intervals use a percentile bootstrap that resamples students as whole clusters (B=5,000): every selected student contributes all of their enrollments. Computationally intensive top-k contrasts and all-model figure intervals use B=2,000. Comparisons between models are paired by applying the same resampled student clusters to both prediction vectors. This matters because the two models’ AUC estimates are strongly correlated (ρ=0.937 for the two ensembles), and an unpaired procedure would inflate the interval for their difference roughly fourfold. As an analytic cross-check, the two focal AUC contrasts were repeated with DeLong’s test for correlated ROC curves (Section 4.8). DeLong treats enrollment rows as independent and is therefore secondary to the student-cluster bootstrap. Comparisons between disjoint subgroups resample student clusters independently within each subgroup.

Multiple comparisons. The subgroup analysis forms a family of 18 pairwise contrasts (two models × five attributes). Two-sided bootstrap p-values use an add-one correction and are adjusted, before rounding, with the Benjamini–Hochberg false discovery rate procedure at q<0.05; both raw and adjusted values are reported. The leave-one-module-out analysis is treated as descriptive: with seven modules, no correction can rescue inference from that few units, so module-level associations are reported as exploratory and hypothesis-generating only. No causal interpretation is made: modules were not assigned to assessment schedules, measured characteristics are mutually confounded, and unmeasured course and cohort differences may account for the observed alignments.

Deployment-oriented metrics. Because alert systems are used to ration tutoring capacity, threshold-free and ranking metrics were computed alongside ROC-AUC for all three HistGB feature sets: average precision, the Brier score, and precision and recall among the 5%, 10%, and 20% of enrollments with the highest predicted risk. Ties at the k-th ranked position are handled explicitly: records scoring strictly above the k-th value are all counted, and the remaining slots are credited at the tied block’s observed hit rate, which is the expected value under random tie-breaking and is invariant to the sort implementation. This is not a pedantic detail—in the behavior-only model, 189 test records share the boundary score at the 5% threshold, filling 166 of 275 available slots, because students with no VLE activity and no submission receive identical predictions.

Temporal validation. To address RQ3 prospectively, the same nine fixed candidate pipelines were compared by five-fold student-grouped cross-validation using 2013 data only. The selected candidate was then fitted to all 2013 presentations (n=11,818; at-risk 44.4%) and tested on all 2014 presentations (n=15,735; at-risk 44.0%). For continuity with the feature-set ablation, all four previously reported temporal pipelines and a like-for-like HistGB ranking comparison are also shown. Students retaking modules appear in both years (1,466 students; 9.3% of 2014 records); because such overlap is part of realistic deployment, the full 2014 cohort is the primary test set, and a sensitivity analysis excluding retakers (n=14,269) is reported alongside. One module (CCC) has no 2013 presentation at all, so for 22.4% of the 2014 test records the temporal split is simultaneously a cross-module extrapolation; results are reported with and without those records.

Cross-module validation and module characteristics. To probe transfer to entirely unseen courses, a prespecified, descriptive leave-one-module-out transport analysis was run: for each of the seven modules, HistGB models (all features and behavior-only) were trained on the other six modules and tested on the held-out module, with a sensitivity variant excluding test-module students who also appear in the training modules. The HistGB configuration was not reselected inside each LOMO fold, so this is not a fully nested deployment estimate. To move beyond attributing the resulting variation to unmeasured “course differences,” each module was additionally characterized by observable properties: subject domain as classified in the original data descriptor (Kuzilek et al., 2017, Table 1) number of presentations, mean presentation length, the number and timing of assessments falling inside the four-week window, the share of enrollments with an early submission, median first-four-week clicks and active days, the share with no VLE activity, and the at-risk base rate. These seven module-level observations are used only to describe candidate explanations for transfer heterogeneity; they cannot estimate the causal effect of assessment timing, domain, or any other module characteristic.

Separating withdrawal from failure. Because merging withdrawal and failure into one label may obscure distinct predictive profiles, three additional prediction tasks were run on the same sample and the same student-grouped split: withdrawal vs. the rest; failure vs. the rest; and, restricted to students who did not withdraw, failure vs. pass or distinction. All withdrawal analyses concern subsequent withdrawal among enrollments active at day 28. Permutation importance was computed for each task so that the relative weight of assessment-based and VLE-based signals could be compared across outcomes.

3.5 Reproducibility

Permutation feature importance was computed on a student-grouped validation fold held out inside the training set (main task nrepeats=30; outcome-specific tasks nrepeats=20), avoiding reuse of the test set. All analyses were scripted; every number reported below is written by the analysis code to labeled output files, and the manuscript was checked programmatically against those files. A separate verification implementation recomputes the principal quantities by different routes—ROC-AUC from the Mann–Whitney U statistic, top-k metrics through a separate sorting implementation, false discovery rates against statsmodels, and the sample flow from a different aggregation—and reports any disagreement. The complete pipeline—feature construction, experiments, the experiment ledger, verification, and step-by-step reproduction instructions—is provided as Supplementary Material.

4 Results

4.1 Predictive performance and feature-set ablation (RQ1, RQ2)

Table 3 and Figure 1 present the full ablation. The cross-validated best configuration was HistGB with all features (R9), reaching a test ROC-AUC of 0.7747 (student-cluster 95% CI 0.7624–0.7870), balanced accuracy 0.6974, and at-risk recall 0.6116.

Table 3. Full ablation matrix under the student-grouped split.

CV: five-fold student-grouped cross-validation on the training set (mean ± SD); test metrics from a single final evaluation on the held-out test set (n=5,518). Recall is for the at-risk class at threshold 0.5. The best row by CV AUC is bolded.

Regarding RQ1, early behavior alone carried substantial signal: with only first-four-weeks behavioral features, HistGB reached a test AUC of 0.7504 (R8), well above both chance level and every background-only baseline (Table 3).

Regarding RQ2, the incremental value of behavioral data held under every classifier and was statistically reliable. Under HistGB, the paired-bootstrap AUC difference between behavior-only and background-only models had a 95% CI of [0.0953, 0.1347], and between all-features and background-only models [0.1239, 0.1551]; neither interval contains zero. Behavior-only models outperformed background-only models under all three classifiers (logistic regression 0.7194 vs. 0.6379; random forest 0.7457 vs. 0.6253; HistGB 0.7504 vs. 0.6352). Conversely, dropping the background/administrative set from the full model cost 0.0243 AUC (R9 vs. R8; paired student-cluster bootstrap 95% CI 0.0165–0.0321). Ensemble tree models outperformed logistic regression (all-features AUC 0.7760 and 0.7747 vs. 0.7447), while no reliable difference was detected between the two ensembles (Section 4.8).

4.2 Feature importance

Table 4 lists permutation importances for the best model. Early mean assessment score was the strongest single signal (mean AUC drop 0.0868, roughly four times the second-ranked feature); this feature combines information about score level with information about whether any submission was made. Among purely behavioral features, active days (0.0112) and week-4 clicks (0.0099) ranked highest, suggesting that the regularity and recency of engagement matter more than raw click volume. Because the click-based features are highly correlated, permutation importance is diluted across them, and within-table ranks should be interpreted with caution.

Table 4. Permutation importance of the best model (R9), computed on a student-grouped validation fold inside the training set (nrepeats=30).

“AUC drop” is the mean decrease in validation ROC-AUC when the feature’s values are randomly permuted, so larger positive values indicate greater importance and a value near zero indicates that the feature carries no information the other features do not already supply. Top five features shown; click-based features are highly correlated, so importance is diluted across them and ranks should be interpreted with caution.

4.3 Ranking utility and the operational cost of data minimization

Alerts are consumed as ranked lists, so Table 5 reports threshold-free and top-k metrics for all three HistGB feature sets, under both the in-pool grouped split and the 2013→2014 temporal split. In-pool, the all-features model achieved average precision of 0.7457 and a Brier score of 0.1898 against a base rate of 0.4409; among the 5%, 10%, and 20% of test enrollments with the highest predicted risk, the proportion truly at risk was 0.9382, 0.9056, and 0.8205, capturing 10.6%, 20.5%, and 37.2% of all at-risk students, respectively.

Table 5. Ranking and calibration metrics for the three HistGB feature sets, under the in-pool student-grouped split (n=5,518; base rate 0.4409) and the 2013→2014 temporal split (n=15,735; base rate 0.4398).

Precision is the share of flagged enrollments that were truly at risk; recall is the share of all at-risk enrollments captured. Ties at the k-th position are credited at the tied block’s hit rate (expected value under random tie-breaking). Because the list length k and the number of at-risk enrollments P are both fixed within a split, recall is precision rescaled by k/P; the two Δ rows per split therefore carry the same information expressed in the two units practitioners use—the purity of the alert list and the share of at-risk students it reaches—and their intervals agree to within rounding.

The comparison across feature sets qualifies the aggregate performance cost reported above. Removing the background/administrative set reduced AUC by only 0.0243, but its effect on the top of the ranking was proportionally larger: in-pool top-5% precision fell from 0.9382 to 0.8655 (Δ=0.0727, 95% CI 0.0339–0.1098), top-10% precision from 0.9056 to 0.8403 (Δ=0.0653, 95% CI 0.0317–0.0949), and top-20% precision from 0.8205 to 0.7643 (Δ=0.0562, 95% CI 0.0356–0.0800). Expressed as coverage rather than purity, the same three contrasts are Δ recall of 0.0082 (95% CI 0.0038–0.0125), 0.0148 (0.0072–0.0217) and 0.0255 (0.0162–0.0365): at a fixed list of the highest-risk 20%, the behavior-only model reaches 2.6 percentage points fewer of the cohort’s at-risk students. Under temporal transfer the gap at the very top widened sharply: top-5% precision was 0.7455 for all features and 0.5687 for behavior only (Δ=0.1768, 95% CI 0.1425–0.2107), while at top-20% the two models were much closer (0.7261 vs. 0.6924; Δ=0.0337, 95% CI 0.0216–0.0441), a coverage penalty at that list length of Δ recall 0.0153 (95% CI 0.0098–0.0200), or 1.5 percentage points of the at-risk cohort. Both models remained far ahead of the background-only baseline, whose top-20% precision was 0.6029 in-pool and 0.5942 under transfer, barely above the base rate.

One pattern in Table 5 is anomalous and consequential: under temporal transfer the behavior-only model’s precision rises with k (0.5687 at 5%, 0.6217 at 10%, 0.6924 at 20%), the opposite of the expected monotone decline. Section 4.6 traces this to cohort drift in the assessment-score signal.

4.4 Separating withdrawal from failure

Table 6 reports the three additional outcome definitions. Predicting subsequent withdrawal against all other outcomes was distinctly harder than the combined task (AUC 0.6971, 95% CI 0.6789–0.7150, vs. 0.7747), as was predicting failure against all other outcomes (0.7200, 95% CI 0.7050–0.7349). Restricting attention to students who did not withdraw, and predicting failure against pass or distinction, was by contrast the easiest of the four tasks (0.7828, 95% CI 0.7676–0.7972). AUCs across different targets are not directly interchangeable, but the results show that the composite outcome does not simply inherit an easier one-vs.-rest task.

Table 6. Separating the composite outcome.

All tasks use the same analytic sample and the same student-grouped split; task D is restricted to enrollments that did not end in withdrawal. “Signal share” gives the proportion of total positive permutation importance contributed by assessment-derived features (early mean score, number of early submissions), by VLE engagement features, and by background/administrative features, for the all-features model of each task.

The predictive signal profiles differ across outcome definitions. For withdrawal, the dominant features were early mean assessment score (AUC drop 0.1014) and credits studied (0.0482), with click-based features contributing little; assessment-derived features accounted for 50.2% of total positive permutation importance and VLE engagement features for 19.2%. For failure against the rest, the ordering reversed: active days was dominant (0.0571) ahead of early mean score (0.0332), with VLE engagement features accounting for 51.4% of importance and assessment features 20.5%. Because the early-score feature is missing precisely when no assessment was submitted, its prominence for withdrawal may partly reflect non-submission as well as score level. Permutation importance does not identify the underlying mechanism or prescribe an intervention.

4.5 Temporal and cross-module validation (RQ3)

Table 7 reports the year-to-year transfer. Grouped cross-validation confined to 2013 selected the random-forest all-features candidate (CV AUC 0.7738±0.0047); fitted on all 2013 data, it reached AUC 0.7470 in 2014 (student-cluster 95% CI 0.7393–0.7549). The HistGB all-features and logistic candidates reached 0.7352 and 0.7217, respectively. The behavior-only HistGB model retained an AUC of 0.7090 (95% CI 0.7009–0.7173). Excluding the 9.3% of 2014 records belonging to retaking students changed results only marginally (e.g., 0.7407 for the random forest; 0.7040 for behavior-only HistGB), indicating that transfer performance is not an artifact of repeat students. Excluding module CCC, which has no 2013 presentation and therefore makes the temporal split partly a cross-module one, moved the estimate from 0.7352 to 0.7208 for the all-features model and from 0.7090 to 0.6878 for the behavior-only model—that is, slightly downward, so the reported temporal figures are not inflated by that module. These comparisons concern the shifts examined here and do not establish performance in other institutions.

Table 7. Temporal validation: models trained on all 2013 presentations (n=11,818) and tested on all 2014 presentations (n=15,735).

The sensitivity column excludes the 1,466 retaking students (9.3% of 2014 records). T2 was selected by grouped cross-validation using 2013 data only. Recall is for the at-risk class at threshold 0.5.

Boldface identifies the model selected by grouped cross-validation using 2013 data only (T2) and its corresponding results.

Transfer to entirely unseen modules was markedly more variable (Table 8): leave-one-module-out AUCs for the all-features model ranged from 0.6426 to 0.7881 (median 0.6990), with the weakest transfer for modules GGG, EEE, and BBB and the strongest for CCC and DDD. Excluding test-module students who also appear in the training modules changed these figures only modestly (e.g., CCC: 0.7881 to 0.7668), so the variability reflects between-module differences rather than student overlap.

Table 8. Leave-one-module-out validation against observable module characteristics.

HistGB models were trained on six modules and tested on the held-out module. Domain is as classified in Table 1 of Kuzilek et al. (2017). “Due <28” is the mean number of graded assessments due before day 28 per presentation; “First day” is the median day of the module’s first assessment; “Subm. %” is the share of enrollments with at least one submission before day 28; “Clicks” is the median cumulative VLE clicks in the first four weeks. The two modules with no assessment due inside the prediction window (EEE, GGG) are also the two with the weakest transfer. With seven modules, associations between these characteristics and transfer are exploratory and non-causal; they may reflect confounding by domain, engagement, cohort composition, or unmeasured course design.

Table 8 pairs those AUCs with observable module characteristics, and one descriptive alignment stands out. The two modules with the poorest transfer, GGG (0.6426) and EEE (0.6827), are also the two with no graded assessment falling inside the four-week window: only 1.0% and 7.4% of their enrollments have any early submission, against 86.7%–92.8% in the other five modules, and their first assessment falls at median day 61 and day 33 respectively. In these modules, early assessment performance—the model’s single strongest input in the pooled analysis—is largely unavailable. Across the seven modules the number of in-window assessments correlates with transfer at Spearman ρ=0.80 (p=0.032), and the at-risk base rate at ρ=0.75 (p=0.052); with n=7 modules these coefficients are unstable and uncorrected, and are reported as exploratory. They do not show that changing an assessment timetable would change transfer performance: domain, timetable, engagement, cohort composition, and unmeasured course properties are confounded. Subject domain is partly confounded with assessment structure: the four STEM modules transferred better on average than the three Social Sciences modules (mean AUC 0.7555 vs. 0.6754), but the STEM module lacking an early assessment (EEE) transferred no better than the Social Sciences modules, and module GGG combines the latest first assessment with the lowest median activity (48 clicks) and the highest share of wholly inactive students (11.1%).

4.6 Cohort drift in the assessment signal

The non-monotone top-k pattern noted in Section 4.3 has a specific and instructive cause, established by stratifying the 2014 cohort by early-submission status (Table 9). For students with no early submission, the behavior-only model trained on 2013 showed almost no calibration-in-the-large error in 2014 (mean predicted risk 0.5424 vs. actual 0.5428), just as it was in 2013 itself (0.5652 vs. 0.5671). For students who submitted but scored in the bottom decile, the relationship broke down: their actual at-risk rate fell from 0.7659 in 2013 to 0.5794 in 2014, while the model continued to assign them a mean predicted risk of 0.9043—a calibration error of +0.3249. Because this group receives the model’s highest scores, it dominates the top of the ranked list and depresses top-5% precision, while the top-20% band reaches down into the well-calibrated non-submitters and recovers.

Table 9. Cohort drift in the assessment signal.

The behavior-only HistGB model was trained on 2013 presentations and applied to 2014 presentations; rows stratify students by early-submission status. Calibration error is mean predicted risk minus actual at-risk rate, so positive values indicate over-prediction. The relationship between non-submission and risk is stable across cohorts; the relationship between low marks and risk is not.

An initial hypothesis that the anomaly was caused by wholly inactive students being over-predicted was tested and rejected: that group’s actual 2014 at-risk rate (0.8704) exceeded its predicted risk (0.7990). For reference, flagging every student with no early submission—a rule requiring no model at all—flags 27.4% of the 2014 cohort at a precision of 0.5428 and a recall of 0.3379, which the model’s top-20% list improves upon (precision 0.7261, recall 0.3302 at a comparable list size).

The diagnosis depends on neither the decile cut nor the model’s random seed. Widening the low-score stratum shrinks the calibration error monotonically (from +0.5270 for the bottom 5% of scores to +0.1398 for the bottom 25%), so the bottom-decile definition used here sits in the conservative middle of that range rather than at its extreme. Across five model seeds the low-score stratum’s calibration error is +0.3129±0.0112 while the non-submitters’ is −0.0048±0.0042, and the top predicted-risk decile of the behavior-only model is over-predicted by +0.2594±0.0123 against +0.1647±0.0216 for the all-features model.

4.7 Subgroup performance, and why privacy is not fairness

Because the privacy argument for behavior-only models would be hollow if their accuracy were uneven across student groups, test-set performance was disaggregated by gender, declared disability, IMD band, age band, and highest prior education, the last two serving as the closest available proxies for the socioeconomic and prior-attainment dimensions that a fairness audit would ideally cover (Table 10).

Table 10. Subgroup performance on the held-out test set for the behavior-only (R8) and all-features (R9) HistGB models.

AUC carries bootstrap 95% confidence intervals. “Flag %” and “Recall@20%” describe a fixed operating point at which the globally highest-risk 20% of enrollments are flagged: the share of the subgroup flagged, and the share of the subgroup’s at-risk students captured. IMD bands are aggregated into thirds; subgroups with fewer than 100 test records are omitted. Across the 18 pairwise contrasts in this family, only the gender gap under R8 survives Benjamini–Hochberg correction (q=0.0144); all other contrasts have q≥0.0612. Intervals and tests resample students as clusters.

Of the 18 pairwise contrasts in this family, 2 reached p<0.05 before correction and 1 survived Benjamini–Hochberg adjustment: the gender gap in the behavior-only model, where AUC was 0.7225 for female and 0.7766 for male students (Δ=−0.0541, 95% CI [−0.0813,−0.0278], q=0.0144). The corresponding gap in the all-features model (Δ=−0.0349, 95% CI [−0.0599,−0.0102], raw p=0.0068) did not survive correction (q=0.0612). The disability gap reported in the previous version of this analysis (Δ=0.0369 for the behavior-only model) is not statistically reliable once multiplicity is accounted for (q=0.4517), and no IMD, age, or prior-education contrast approached significance.

The direction of the surviving gap is the substantive point. Removing the background/administrative features did not reduce subgroup disparity: across the nine subgroup contrasts, the absolute AUC gap narrowed in only two, and for the one reliable gap it widened, by 0.0192 (paired student-cluster bootstrap 95% CI 0.0042–0.0344). Because this particular comparison is the one most likely to be an artifact of a single data split, it was recomputed across ten repeated student-grouped random splits (Section 4.9): removal widened the gender gap in nine of them, by 0.0093 to 0.0331, and narrowed it in the tenth by 0.0012—that is, not at all in practical terms. We therefore claim only that removing the background set does not reliably reduce this disparity, not that it reliably increases it. Behavior-only models can retain subgroup performance disparities without explicit group labels; this analysis does not establish how much group membership is encoded in the behavioral predictors.

At a fixed operating point the picture is more nuanced, and the two readings should not be conflated. Applying a global top-20% threshold, the all-features model flags 31.3% of students declaring a disability against 18.8% of those not declaring one, and 29.4% of the most-deprived IMD tercile against 14.4% of the least-deprived; the behavior-only model flags these groups more evenly (25.7% vs. 19.4%; 24.8% vs. 16.5%). Background-feature removal thus moves the system toward equal flag rates. But because the observed at-risk prevalence was higher in those groups—52.1% and 52.5%, respectively—the same change lowers how many high-risk students in them the system reaches: recall within the most-deprived tercile falls from 0.4639 to 0.3818, and within the disability-declaring group from 0.4607 to 0.3643. Whether that is an improvement depends on which fairness criterion an institution adopts. Background/administrative-feature removal showed no consistent improvement on the audited AUC, flag-rate, and recall criteria; this is not a comprehensive assessment of fairness. Accordingly, excluding these inputs from the scoring model is a data-minimization/privacy choice, whereas fairness requires a separate subgroup audit and cannot be inferred from feature removal.

4.8 Bootstrap pairing and analytic cross-check

Because the conclusion that HistGB and random forest do not reliably differ rests on the comparison procedure, Table 11 reports it three ways. The paired bootstrap gives an AUC difference of −0.0013 with a 95% CI of [−0.0057,0.0031]; DeLong’s test for correlated ROC curves gives [−0.0056,0.0030] with z=−0.59 and p=0.554. The two agree to within 0.0001 AUC. Had the resampling been unpaired—each model resampled independently—the interval would have been [−0.0191,0.0163], 4.02 times wider, because it would discard the strong correspondence between the two models’ predictions. The same three procedures applied to the background-set contrast (R9 minus R8) give [0.0165,0.0321] paired, [0.0168,0.0319] by DeLong (z=6.345,p<0.001), and [0.0064,0.0421] unpaired. All primary intervals reported in this article for model-vs.-model contrasts use paired student clusters; DeLong ignores within-student dependence and is only a sensitivity check.

Table 11. Model-vs.-model comparison under three procedures, on the held-out test set (n=5,518; 4,970 students).

The paired bootstrap applies the same resampled student clusters to both models; the unpaired variant resamples clusters independently and is shown only as a counterfactual. DeLong’s test provides a row-independence analytic sensitivity check. ρ is the correlation between the two models’ AUC estimates.

4.9 Sensitivity and robustness analyses

Three prediction-time availability concerns were tested directly. First, because OULAD does not record when a submitted assessment was marked or its score released, early_mean_score was removed while submission count and all VLE features were retained. In-pool AUC declined from 0.7747 to 0.7381 for all features (loss 0.0366; student-cluster 95% CI 0.0285–0.0449) and from 0.7504 to 0.7027 for behavior only (loss 0.0477; 95% CI 0.0379–0.0573). Under temporal transfer the corresponding no-score AUCs were 0.7050 and 0.6684. Notably, removing the drift-prone score improved temporal top-5% precision from 0.7455 to 0.9135 in the all-features HistGB model, even while overall AUC fell. The score-inclusive results should therefore be read as conditional on timely score release; the strict no-score models still retain useful, but weaker, discrimination.

Second, excluding the 18 enrollments registered on or after day 28 left performance essentially unchanged (all-features AUC 0.7781; behavior-only 0.7519, vs. 0.7747 and 0.7504 in the main analysis). Third, recomputing early assessment features with banked scores removed and refitting the models also changed results negligibly (all-features AUC 0.7767; behavior-only 0.7515). The main conclusions are therefore robust to the late-registration and banked-score concerns.

Robustness to the choice of split. Every in-pool result above rests on one student-grouped split, so the findings new to this analysis were recomputed across ten repeated student-grouped random splits using different seeds (Table 12). The cost of removing the background/administrative feature set was positive in all ten splits on all four metrics: 0.0243–0.0326 in AUC, 0.0504–0.1127 in top-5% precision, 0.0418–0.0759 at top-10%, and 0.0344–0.0618 at top-20%. The split used for the main analysis returns the smallest AUC cost of the ten, so the headline figure understates rather than flatters the effect. The gender gap was negative—lower discrimination for female students—in every split, for both the behavior-only model (−0.0305 to −0.0703) and the all-features model (−0.0163 to −0.0553).

Table 12. Robustness of the findings new to this analysis across ten repeated student-grouped random splits using different seeds.

“Main split” is the seed used throughout the rest of the article. “Consistent” indicates whether the quantity kept the same sign in all ten splits. The cost of removing background/administrative features is positive everywhere and is smallest in the main split; the gender gap is negative everywhere; only the change in that gap when the background set is dropped fails the consistency test, and the text is worded accordingly.

The one quantity that is not fully stable is the change in that gap when the background/administrative set is dropped: removal widened the gap in nine splits (0.0093–0.0331) and narrowed it by 0.0012 in the tenth. The claim made in Sections 4.7, 5.3 is worded accordingly—that data minimization does not reliably reduce the disparity, not that it reliably increases it. Under the temporal split, which is determined by the data and admits no reseeding, five model seeds gave AUC 0.7071±0.0017 (behavior-only) and 0.7352±0.0012 (all features), confirming that the year-to-year results are not sensitive to model initialization either.

5 Discussion

5.1 How much early warning is possible by week four? (RQ1)

Under a protocol designed to limit leakage—student-grouped splits, cross-validation-only model selection, leakage-free imputation—the achievable discrimination at the end of week four on OULAD is an AUC of about 0.77, and about 0.75 from behavior alone. These figures are deliberately early, but they assume that scores attached to pre-day-28 submissions are available by the prediction point. Removing scores reduces AUC to 0.738 with all remaining features and 0.703 from behavior alone, providing a strict availability benchmark. The estimates are not directly comparable to the higher accuracies often reported on OULAD with full-course data or other protocols (Waheed et al., 2020; Adnan et al., 2021; Borna et al., 2024): prediction naturally becomes easier as the course progresses (Adnan et al., 2021), and record-level splitting would have placed students from 15.7% of test records in both partitions. The resulting change in predictive performance was not estimated directly. The leakage literature suggests that such protocol differences can account for substantial performance gaps (Kapoor and Narayanan, 2023; Apicella et al., 2025); quantifying them on a public benchmark is one contribution of this study. That a four-week window already supports AUC≈0.75--0.77 is consistent with early Open University results showing that clicking behavior combined with assessment scores identifies failing students well before course end (Wolff et al., 2013), with studies that deliberately restrict themselves to early, limited data (Azizah et al., 2024), and with findings elsewhere that early academic performance dominates dropout prediction in the first weeks (Gonzalez-Nucamendi et al., 2023).

5.2 Behavior vs. background, and the price of data minimization (RQ2)

The head-to-head ablation yields a clear ordering: what students do in the first four weeks predicts risk substantially better than who they are at enrollment, under every classifier tested. This echoes evidence that LMS activity and early grades carry the bulk of predictive signal (Macfadyen and Dawson, 2010; Tempelaar et al., 2015; Conijn et al., 2017), and complements OULAD-specific analyses of demographic effects (Rizvi et al., 2019).

The predictive-performance cost of forgoing the full background/administrative set, however, should be quoted in the currency in which alerts are spent. In aggregate discrimination it is small: 0.0243 AUC in-pool (95% CI 0.0165–0.0321), with temporal transfer holding at 0.7090. In ranking terms it is larger and rises towards the top of the list, which is exactly where a capacity-constrained tutoring service operates: the top-5% precision penalty is 0.073 in-pool and 0.177 under year-to-year transfer. An institution that can contact only the highest-risk 5% of a cohort therefore pays materially more for data minimization than the AUC difference implies, whereas one that works through the top 20% pays little (0.034 under transfer). We accordingly retain the finding that background-free risk ranking is technically feasible on OULAD, but not the framing of a straightforward “privacy dividend”: the trade is real, its size depends on where the alert threshold sits, and it should be measured at the operating point an institution actually intends to use. Because the omitted set contains three prior-study/registration variables as well as six demographic or socioeconomic attributes, this contrast is not an isolated estimate of the effect of removing demographic variables alone.

Compliance with data-minimization principles (Drachsler and Greller, 2016; Pardo and Siemens, 2014) thus becomes a defensible design choice rather than a costless one. A behavior-only model is also easier to explain to students and staff, because its inputs are actionable: logging in regularly and submitting early assessments are behaviors a student can change, whereas an IMD band is not, and students’ own accounts suggest they distinguish sharply between analytics that support learning and analytics that feel like surveillance (Soffer and Cohen, 2024; Francis et al., 2023). We note, finally, that “privacy-friendly” here refers specifically to forgoing demographic profiling under data-minimization principles; the collection of behavioral telemetry itself raises surveillance and consent concerns that institutional policy must still address (Slade and Prinsloo, 2013; Drachsler and Greller, 2016; Prinsloo and Kaliisa, 2025).

5.3 Privacy is not fairness

The subgroup results in Section 4.7 force a distinction that the data-minimization argument tends to blur. Reducing the demographic data an institution collects is a privacy intervention: it lowers exposure, narrows the purposes to which records can be repurposed, and shortens the chain of consent that must be maintained. Reducing disparity in how well a model serves different groups is a fairness intervention, and the two do not follow from one another. In these data, removing the nine background/administrative features left the disability, deprivation, age, and prior-education contrasts statistically indistinguishable, narrowed the AUC gap in only two of nine contrasts, and widened the single gap that survives correction for multiplicity—the gender gap—in nine of ten repeated random splits, by 0.009 to 0.033 AUC, narrowing it in the tenth by 0.001. The defensible reading is not that data minimization makes fairness worse, but that it does not make it better. Behavioral traces are not neutral: patterns of when and how much students engage are themselves shaped by employment, caring responsibilities, digital access, and prior educational experience, so they carry group information whether or not the group label is supplied (Baker and Hawn, 2022; Opoku et al., 2025).

The operating-point analysis adds a second layer. Background/administrative-feature removal made flag rates more even across groups but reduced the share of observed at-risk students reached within the highest-risk groups. Equal treatment and equal benefit pull in opposite directions here, which is precisely the trade-off that recent work on accuracy–fairness tensions in learning analytics documents (Opoku et al., 2025), and which early warning systems evaluated only on aggregate metrics will not reveal (El Jihaoui et al., 2026; Xia and Chen, 2026). The practical recommendation is therefore not “drop demographics and fairness follows” but “drop demographics for privacy reasons if the accuracy price is acceptable, and audit subgroup performance separately, at the operating point, before and after deployment.” Retaining demographic attributes for audit while excluding them from the model is a coherent middle course, subject to an appropriate legal basis, purpose limitation, access controls, and local governance.

5.4 Transfer across years and modules, and descriptive correlates (RQ3)

Grouped model selection conducted within 2013 chose the random forest, whose 2014 AUC was 0.7470. The other fixed candidates were lower by varying amounts, but pairwise temporal uncertainty was not assessed. Transfer to entirely unseen modules was far less predictable: leave-one-module-out AUCs ranged from 0.64 to 0.79, a spread wider than any other protocol difference examined in this study.

The module-level analysis in Table 8 does not explain this spread causally; it identifies descriptive correlates that can motivate tests in larger, prospectively designed studies. The most prominent alignment concerns assessment structure: the two modules without a graded assessment inside the four-week window are also the two with the weakest all-features transfer. Both models performed weakly in EEE and GGG; behavior-only AUC was 0.6173 in EEE and 0.6464 in GGG. Subject domain aligns only partly—STEM modules transferred better on average, but the STEM module without an early assessment did not. Module GGG also combines the latest first assessment with the thinnest behavioral signal (median 48 clicks; 11.1% of students wholly inactive). These features are confounded across only seven modules, and unmeasured differences in course design or cohort composition may produce the same pattern. The analysis is therefore exploratory and non-causal: it supports the testable hypothesis that the availability of in-window assessment information affects portability, but it does not establish that changing an assessment timetable would improve transfer. Operationally, institutions should check whether the target module offers the inputs used by a pooled model and validate performance locally; weaker transfer where inputs differ is a possibility to test, not a presumed effect.

Together, the two analyses quantify, on a public benchmark, the portability concern raised by multi-course studies (Conijn et al., 2017) and by critiques of one-size-fits-all analytics (Gašević et al., 2016): course content, cohorts, and platform use can shift over time, so in-pool estimates may overstate performance under the shifts examined here. The result also supports the practice, adopted by production systems such as OU Analyse, of validating on future presentations and continuously retraining (Herodotou et al., 2019; Hlosta et al., 2017). Part of the observed year-to-year decline may reflect the smaller training set available under the temporal split (n=11,818 vs. 22,035) and differences in module–presentation composition between years rather than temporal drift alone; the observed decline cannot be attributed to temporal drift per se. We recommend that OULAD studies report temporal and cross-module splits alongside any random split as a matter of routine.

5.5 Withdrawal and failure: an additional outcome analysis

The outcome-specific models showed different predictive signal profiles. Among enrollments active at day 28, subsequent-withdrawal predictions relied more on assessment non-submission information and credits studied, whereas failure predictions relied more on VLE engagement. These associations do not identify causal mechanisms or prescribe interventions. Moreover, AUCs across tasks with different targets and negative classes are not directly comparable, so they do not establish that nothing is lost by using the composite outcome. Instead, the findings motivate prospective evaluation of differentiated support pathways.

5.6 Implications for practice

The results support a prospective, locally validated pilot of a week-four risk-ranking component using routinely collected data and without including the background/administrative feature set in the scoring model as a data-minimization/privacy choice, not as a fairness intervention. They do not establish the effectiveness of a deployed early warning system. Four caveats shape such a pilot. First, at the default threshold of 0.5 the temporal models recall 57%–59% of at-risk students; thresholds should be tuned to intervention capacity, or students ranked by risk score and tutoring resources allocated top-down (Jayaprakash et al., 2014; Foster and Siddle, 2020)—a strategy the top-20% analysis supports, with roughly three of four flagged students truly at risk even under temporal transfer. Second, before deploying the model to each new presentation, calibration should be assessed using out-of-sample predictions and mature outcomes from comparable completed presentations. Operationally, this means retaining the trained ranking model and using archived scores and outcomes from the most recent completed presentation(s), preferably of the same module, to assess calibration-in-the-large, calibration slope, Brier score, and precision at the planned alert-list size, both overall and in prespecified score/submission strata. If those checks show drift, a prespecified probability-calibration mapping (for example, logistic or isotonic calibration) should be fitted to those held-out completed cohorts, and the intervention threshold should then be fixed before scoring the new cohort. If a monotone calibration map cannot repair a change in ranking, the underlying model must instead be refitted on a rolling set of completed cohorts and revalidated. Outcomes from the cohort being predicted must not be used until they mature; they inform calibration for the next presentation. Recalibration is thus distinct from retraining the underlying model or tuning a threshold after seeing current outcomes. Where too few comparable completed cases exist, the uncertainty should be reported and the model locally validated before its probabilities trigger high-stakes action. This recommendation follows from Section 4.6: the relationship between low early marks and eventual risk shifted materially between consecutive academic years, while the relationship between non-submission and risk did not. Third, prediction is only the first half of an early warning system; evidence on the effectiveness of the interventions triggered by alerts remains limited (Ifenthaler and Yau, 2020), and teacher uptake is decisive in practice (Herodotou et al., 2019; Xia and Chen, 2026). Fourth, the dominant predictor—early assessment performance—is usable only when scores are released by day 28. A strict score-free specification remains available when that condition is not met, although its discrimination is lower. Assessment performance is itself partly an engagement signal (Fredricks et al., 2004; You, 2016): non-submission by week four is a self-evident warning sign, and the appropriate response is pedagogical (contact, support, flexible deadlines) rather than merely statistical.

Taken together, responsible implementation requires a staged pathway: first, establish predictive feasibility under the score-availability conditions of the target setting; second, conduct prospective local validation; third, audit calibration and subgroup performance at the planned operating point; fourth, run a controlled pilot with a specified support protocol and governance; fifth, continuously monitor ranking performance, calibration, flag rates, subgroup performance, intervention uptake, and student outcomes; and sixth, recalibrate using out-of-sample predictions and mature outcomes from completed cohorts, or retrain and revalidate when the ranking or predictive structure changes materially. These steps are safeguards requiring prospective validation, not evidence that deployment has already improved outcomes.

5.7 Limitations

Several limitations should be acknowledged. External validity. Although splitting was grouped by student and validation covered both a year-to-year transition and unseen modules, all analyses remain within a single institution, a single platform, and a single national context; performance in other universities, countries, disciplines, or learning management systems may differ. The exploratory module-level pattern suggests assessment-timetable differences as one hypothesis for future testing, not as an established cause of transfer variation. Interpretation of behavioral features. Behavioral variables should not be read as neutral measures of effort: engagement patterns are shaped by employment, caring responsibilities, digital access, and prior educational experience, which is why they retain group information after the background/administrative features are removed. Outcome definition. Withdrawal and failure were merged in the primary analysis; Section 4.4 shows different predictive signal profiles, but cannot compare the utility of the composite and outcome-specific tasks or identify causal mechanisms (Tinto, 1975). Scope of the fairness audit. The audit covers subgroup AUC disparities for the attributes and contrasts specified here; it does not establish equalized odds, calibration parity, individual fairness, or the absence of intersectional harms. Age band and highest prior education were added in this revision as proxies for socioeconomic background and prior attainment, but direct measures of income, employment status, caring responsibilities, and digital access are unavailable, as are intersectional breakdowns, which the subgroup sizes here would not support. Statistical scope. Module-level associations rest on seven units and are exploratory and non-causal; assessment timing, domain, engagement, cohort composition, and unmeasured course design cannot be disentangled. The subgroup analysis is corrected for multiplicity within its family but the family was specified after the main analyses were complete. Intervention scope. No alert, support protocol, or controlled deployment was evaluated; predictive discrimination and ranking utility therefore do not establish intervention effectiveness or improved student outcomes. Modeling scope. The early-assessment feature was filtered by submission date, but in deployment grades may not yet be released at day 28. The strict score-free sensitivity analysis therefore provides the appropriate benchmark for such settings and shows a material discrimination loss; banked scores and late registration are separate concerns that were tested directly (Section 4.9). The models capture association, not causation, and permutation importances of correlated click features are diluted, so they should not directly dictate intervention design. Finally, logistic regression was used with default regularization and standard preprocessing only, so the gap to tree ensembles should not be read as strict evidence of nonlinearity.

5.8 Future work

Natural extensions include intersectional fairness audits with formal criteria and adequate subgroup sizes (Baker and Hawn, 2022; Opoku et al., 2025), cost-sensitive thresholding aligned with tutoring capacity, sliding prediction windows that update weekly, explicit recalibration schemes for the score-based features identified as drift-prone here, replication of the module-characteristics analysis on datasets with more than seven courses, and, most importantly, experimental studies linking model-triggered interventions to retention outcomes (Ifenthaler and Yau, 2020).

6 Conclusion

This study quantified, under a leakage-controlled and fully reproducible protocol, the early warning achievable on the OULAD benchmark at the end of the fourth course week. Behavioral traces from the first four weeks support substantial discrimination of academic risk (AUC 0.7504 behavior-only; 0.7747 with background/administrative features; student-cluster bootstrap 95% CI 0.7624–0.7870). These principal score-inclusive estimates apply only when scores from pre-day-28 submissions have been released by the prediction time; OULAD records submission dates and eventual scores but not the exact marking or score-release timestamps. Where that availability condition cannot be established, the strict score-free all-feature sensitivity (AUC 0.7381) is the appropriate deployment benchmark. When model selection was confined to 2013 presentations, the selected random forest reached AUC 0.7470 (95% CI 0.7393–0.7549) in 2014; the prespecified behavior-only HistGB reached 0.7090. Transfer to entirely unseen modules was markedly less reliable (AUC 0.64–0.79). In the exploratory seven-module comparison, the weakest cases also lacked assessed work inside the prediction window. This descriptive alignment is non-causal and supports local validation and replication, not a claim that changing assessment timing would itself improve transfer. These results establish the feasibility of prediction under the specified conditions only; they do not show that alerts, targeted support, or deployment improve student outcomes.

For research, the study demonstrates how much apparent performance can hinge on evaluation protocol—record-level splitting alone would have contaminated 15.7% of test records—and argues for student-grouped and temporal validation as default practice on educational benchmarks. For practice, it quantifies a data-minimization trade-off rather than merely advocating it: the behavior-only ranking omits six demographic/socioeconomic and three administrative/prior-study inputs at a cost of 0.024 AUC, with a larger and threshold-dependent cost in the precision of the highest-risk band, without a reliable or consistent improvement across the audited subgroup criteria. Privacy and fairness are separate objectives requiring separate instruments, and an institution that excludes demographic attributes from scoring still owes its students a subgroup audit at the operating point it deploys, using separately governed audit data where legally and ethically appropriate. Responsible use therefore requires prospective local validation, calibration and subgroup auditing, controlled deployment, continuous monitoring, and recalibration or retraining from completed cohorts when needed.

All analysis code, labeled intermediate outputs, the experiment ledger, and a separate verification script accompany this article, so that every number reported here can be traced to the line of code that produced it and the full pipeline can be reproduced end-to-end.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2026.1917092/full#supplementary-material.

Supplementary material Supplementary material at the source

Footnotes

Edited by: Glenn Agung Hole, University of South Eastern Norway, Norway

Reviewed by: Amila N. K. K. Gamage, LIGS University, United States

Aida Chefrour, University of Souk Ahras, Algeria

Abbreviations AUC, area under the receiver operating characteristic curve; BH, Benjamini–Hochberg; CI, confidence interval; CV, cross-validation; EWS, early warning system; FDR, false discovery rate; HistGB, histogram-based gradient boosting; IMD, index of multiple deprivation; LMS, learning management system; LOMO, leave-one-module-out; OULAD, Open University Learning Analytics Dataset; AP, average precision; ROC, receiver operating characteristic; SRL, self-regulated learning; VLE, virtual learning environment.