Benchmarking, Evaluation, and Reproducibility in Earth Observation Machine Learning

Earth observation relies on diverse imaging systems whose varying spatial, spectral, radiometric, and temporal characteristics present unique challenges for the development and evaluation of machine learning methods. Recent advances in deep learning, including foundation models, have substantially improved the analysis of EO imagery, but have also exposed fundamental limitations in how imaging-based models are evaluated, compared, and reproduced.Earth observation is, in many cases, inherently an imaging problem. Variations in sensor characteristics, spatial and spectral resolution, acquisition geometry, atmospheric conditions, seasonal effects, and domain shifts across regions introduce complexities that challenge standard benchmarking practices. Dataset construction, annotation uncertainty, and long-standing labeling inconsistencies further complicate the interpretation of reported performance gains. As a result, leaderboard improvements often obscure underlying sensitivity to imaging conditions, sensor-specific biases, and domain transfer limitations.Despite the growing volume of published research, obtaining a clear and reliable view of the state of the art remains difficult. Progress is hindered by the lack of well-established benchmarking protocols, inconsistent or incomplete reporting of experimental results, limited standards for interpretability and error analysis, and insufficient diagnostic evaluation tailored to imaging characteristics. Reported performance gains are therefore often difficult to reproduce, compare across studies, or generalize to new sensors, resolutions, acquisition conditions, and geographic domains.This Research Topic aims to advance imaging science in Earth observation by promoting rigorous, transparent, and reproducible evaluation practices for machine learning models. The focus is on benchmarking, performance analysis, and methodological soundness grounded in the realities of EO imaging systems, rather than on incremental model development. Contributions are expected to provide insight into model behavior under varying imaging conditions, quantify robustness to domain and sensor shifts, and enable reliable comparison across datasets and modalities.We invite submissions of original research articles, large-scale and reproducible evaluation studies, critical analyses of existing benchmarks, and substantial contributions to dataset development or improvement. Submissions that primarily report minor architectural modifications or limited empirical gains without comprehensive evaluation, reproducibility analysis, or imaging-relevant methodological insight are out of scope for this Research Topic.Topics of interest include, but are not limited to:Design and validation of EO imaging benchmarks that account for sensor characteristics, spatial resolution, and acquisition variabilityEvaluation protocols that explicitly address cross-sensor, cross-resolution, and cross-domain generalizationLarge-scale, consistent, and reproducible evaluation of state-of-the-art models under diverse imaging conditionsDiagnostic and interpretable evaluation frameworks that go beyond aggregate accuracy scores, including uncertainty estimation and error stratificationSystematic analysis of existing EO imaging datasets, including re-annotation, bias assessment, label noise quantification, and uncertainty modelingMethods and tools for ensuring dataset quality, consistency, and long-term usability in imaging-centric EO workflowsSubmissions may address specific EO tasks, such as classification, detection, segmentation, or change detection, or adopt a broader, task-agnostic perspective across sensors and imaging modalities. All contributions will be evaluated primarily on scientific rigor, methodological clarity, reproducibility, and their contribution to establishing reliable and imaging-aware evaluation standards in Earth observation. Topic editors: Luka Čehovin Zajc, Jorge Delgado García, Ondrej Krejcar Source: https://www.frontiersin.org/research-topics/80211/benchmarking-evaluation-and-reproducibility-in-earth-observation-machine-learning