HydroTuring: A physics probe benchmark testing mass, energy and momentum conservation in hydrologic models
Zhi Li, Changming Li, Siavash Shams, Han Wang, Qingyi Yang, Yuanhang Liu, Taiqi Lian, Xin Lan, Shunan Zhou, Binlan Zhang, Siddik Barbhuiya, Songkun Yan, Bing Li, Jinlong Hu, Yaji Wang, Zhenjiang Wu, Mofan Zhang, Kaihao Long, Jingzhi Chen, Yuan-Heng Wang
Abstract
Hydrologic models, including deep learning systems now deployed for operational flood forecasting, are evaluated primarily by how closely their simulated hydrographs match observed streamflow. However, high hydrograph skill does not guarantee physical consistency: in a synthetic calibration experiment, a conceptual model that loses 15% of its water input to an unreported sink still reaches a Nash-Sutcliffe efficiency of 0.93, and no shared benchmark tests whether models obey fundamental conservation laws. Here we introduce HydroTuring (v0.1.0), a black-box benchmark of 37 physics probes that test mass (24 probes), energy (7) and momentum (6) conservation in hydrologic models. Each probe runs a controlled experiment on forcing generated at run time from a recorded seed and applies binary pass or fail criteria validated against 18 physically exact and 59 deliberately broken reference models. Across 32 evaluated models (3 exact references, 7 AI-based models and 22 process-based models), only the 3 exact references pass every probe among the 28 models scored on 5 or more probes. Process-based and AI-based models pass similar shares of their scored probes (76% and 74%), and a pre-trained flood-forecasting LSTM fails 6 of the 11 runoff-only probes it can be scored on. Furthermore, 49% of all 1,184 model-probe pairs remain unscored because models omit the required state or flux variables, while the scored failures occur in perturbation responses and state bounds rather than long-term water-budget closure. HydroTuring provides a reproducible smoke test for model development, peer review and operational deployment under an open community protocol.
Review reports
xPeerd (AI review)
AI Review (xPeerd — DAReview)
Summary
This manuscript introduces HydroTuring v0.1.0, a black-box benchmark of 37 “physics probes” designed to test whether hydrologic models satisfy conservation laws for mass, energy, and momentum rather than merely fitting streamflow records. The paper’s core motivation is clear and important: high hydrograph skill can coexist with physically inconsistent behavior, illustrated by a synthetic example where a model that “loses 15% of its water input to an unreported sink” still achieves NSE = 0.93 . The manuscript is strongest where it defines a concrete benchmark architecture, validates probes against exact and deliberately broken references, and reports large-scale comparative results across 32 evaluated models . The contribution is timely and potentially influential for hydrologic model evaluation.
The main weaknesses are not in the motivation but in the evidentiary framing and scope of claims. The benchmark is positioned as broadly relevant to “hydrologic models as a whole,” yet many results are adapter-dependent, many model–probe pairs are unscored, and the synthetic setup constrains ecological validity . Some conclusions are stronger than the evidence strictly supports, particularly where the manuscript compares AI-based and process-based models despite highly uneven observability and probe applicability. Overall, this is a strong methodological paper with real novelty, but it would benefit from tighter claim calibration, clearer treatment of adapter dependence and missingness, and more explicit safeguards against over-interpretation.
Potential Major Revisions
- The manuscript should narrow and qualify several headline claims so they do not overreach the benchmark’s demonstrated scope.
The paper states that “HydroTuring therefore targets hydrologic models as a whole, whether learned, physical or hybrid” and later claims that “HydroTuring provides a reproducible smoke test for model development, peer review and operational deployment” . These statements are ambitious relative to the actual test environment. As the limitations section acknowledges, “the catchment and the weather are synthetic,” “most probes run on 1 invented lumped catchment,” and “a model trained on real basins that misbehaves in the synthetic catchment may violate physics or may operate far outside its training distribution” . This is a serious caveat, not a minor one. The manuscript should more consistently state that the current release demonstrates a controlled synthetic benchmark for physics-related falsification signals, not a comprehensive or deployment-sufficient validation framework.
- The interpretation of comparative performance across model classes requires more caution because the missingness structure is large and non-random.
The abstract reports that “Process-based and AI-based models pass similar shares of their scored probes (76% and 74%)” while “49% of all 1,184 model-probe pairs remain unscored because models omit the required state or flux variables” . Section 4.1 similarly states that “49% of all model-probe pairs remain unscored due to missing output variables or time-step mismatches rather than physical failures” . This level of incompleteness substantially limits cross-class comparison. A streamflow-only model, for example, is N/A on many conservation-related probes by design . Because observability is itself structurally tied to model class and modeling philosophy, pass fractions among “scored probes” are not directly comparable without a more formal missing-data discussion. I recommend adding a dedicated subsection explaining why these percentages should or should not be interpreted comparatively, and ideally stratifying by probe family and output availability.
- The benchmark’s adapter dependence is substantial and should be elevated from a limitation to a central methodological issue.
The paper correctly notes that “a verdict therefore evaluates a model together with its adapter” . This is especially consequential in the LSTM case study, where “catchment attributes are taken from the first year of the generated record or set to the training mean,” and the evaluation “reflects the released weights coupled with this adapter rather than the full operational forecasting system” . Likewise, for the differentiable model, the manuscript attributes a likely cause to calibration/time-step scaling, but that remains inferential . Since adapter choices can alter verdicts, the paper should provide a more systematic sensitivity analysis or, at minimum, a table identifying which reported findings are adapter-robust and which are adapter-sensitive. Without this, some model-level conclusions risk being interpreted as intrinsic failures when they may partly reflect packaging or translation choices.
- The paper should more explicitly separate benchmark validation from scientific claims about model physics.
The reference-model acceptance gate is a strong design feature: “Every probe must therefore pass an acceptance gate… exact reference models… must pass, and deliberately broken reference models… must fail” . However, validation against intentionally engineered references is not equivalent to demonstrating discriminative validity on real hydrologic systems. The benchmark proves that the probes detect designed violations under controlled conditions; it does not yet fully establish that every failed real-model verdict unambiguously corresponds to a meaningful physical inconsistency rather than a representation mismatch. The paper should articulate this distinction more sharply, especially because some failures involve hidden assumptions about compatible state definitions, temporal aggregation, or omitted variables.
- The manuscript’s strongest empirical finding may not be the one it foregrounds, and the discussion should be rebalanced accordingly.
The paper emphasizes that “evaluated models rarely fail on long-term water-budget closure” and that failures instead occur “in the response of the model to a perturbed input and in the physical bounds of reported states and fluxes” . This is a strong and interesting finding. Yet parts of the framing still center heavily on closure-style conservation failures, beginning with the introductory leaky-model parable . Since the actual benchmark results show “Long-term water budget 0 failed pairs (all 24 reporting models pass)” in one summarized view , the paper should emphasize more prominently that the benchmark’s practical value lies less in catching gross whole-record leakage and more in detecting eventwise, perturbational, and bound-related failures. That would better align the narrative with the evidence.
- Several claims about operational or peer-review use are promising but premature without governance details for evaluator consistency.
The paper states that “Journals can also run submitted models through the suite as a reproducibility and consistency check” . This is plausible, but such use would require standardized adapter construction, transparent declaration of unsupported outputs, and rules for handling N/A-heavy models. The governance section is thoughtful, but currently more repository-centered than evaluation-centered . If the paper wants to claim journal- or deployment-level use cases, it should specify minimum reporting requirements for third-party evaluators, including adapter provenance, sensitivity notes, and the proportion of applicable probes.
- The manuscript should discuss whether strict binary verdict aggregation may obscure severity and uncertainty.
The paper adopts a strict rule: “A model passes only when every criterion of every probe that applies to it passes. A single violated law is enough to fail” . This is philosophically coherent, but it may be too coarse for some scientific contexts. The manuscript partially compensates by reporting measured quantities and counts, yet the paper would benefit from a more explicit rationale for how strict pass/fail is intended to be interpreted by users. For instance, is a slight threshold exceedance epistemically comparable to a severe budget contradiction? A short discussion on gradation, severity, and uncertainty would make the framework more defensible for broad use.
- The manuscript should include a clearer novelty comparison against adjacent benchmark traditions.
The paper claims, “To our knowledge, no benchmark in hydrology or in Earth system science tests conservation laws on hidden, generated cases” . This may well be true, but the novelty claim is central enough that it deserves a more explicit comparative synthesis in the related-work section. At present, the manuscript mentions conventional hydrograph metrics and some AI-for-science benchmarks, but the distinction between HydroTuring and related physical-consistency or physics-informed evaluation frameworks could be sharper. This would reduce the risk that the novelty case appears asserted rather than demonstrated.
- The anti-fraud / fabrication screen does not raise major concerns, but a few features merit caution.
I do not see strong paper-mill indicators. The manuscript presents a coherent paradigm, a reproducible artifact trail, explicit limitations, and nontrivial technical structure . That said, some references are unusually current, including multiple 2026 sources and an arXiv preprint date close to the claimed benchmark timeline . This is not evidence of fraud, but because the manuscript’s contribution depends heavily on code, containers, and archived artifacts, the review record should verify that the cited repository, Zenodo archive, and model manifests reproduce the reported counts. The article’s credibility hinges on artifact integrity more than on prose alone.
Potential Minor Revisions
- Typographic, spacing, and line-break issues
The PDF extraction reveals multiple formatting and tokenization defects that should be corrected before submission.
- On page 1, the metadata line appears concatenated as “arXiv:2611.11563v1 [physics.geo-ph] 8 Nov 20261Flood Lab…” indicating a missing space or line break between the date and affiliation block .
- On page 7, “timeorigin-invariance” and “groundwaterdatum-invariance” contain broken character encoding/hyphenation artifacts .
- On page 13, the section/page transition appears merged as “133.4 Reference models and probe validation,” suggesting a page number or layout collision before Section 3.4 .
- On page 18, “18precipitation-counterfactual” appears without a space after the page break marker .
- On page 19, “multiscale differentiable physics-informed machine learning” is split with a broken hyphenation marker as “nationalscale” and similar artifacts appear elsewhere .
- On page 11, the phrase “Shunan Zhou” shows a character encoding issue in the author name .
- On page 3, “2Third” appears as a merged token at the start of a paragraph, indicating a missing line break or page-header artifact .
- On page 12, “39Table 15” indicates another page-number/table-title merge .
- Wording and style issues
- The sentence “We envision that new AI-based and physics-based models will be tested against HydroTuring as a smoke test before they are deployed” is readable, but “envision” plus “will” is rhetorically strong for a tool not yet validated on real-data tracks . Consider softening.
- The phrase “HydroTuring asks whether a model behaves like the physical system under questions that no training record answered” is elegant, but somewhat anthropomorphic and could be tightened for a methods paper .
- The discussion sentence “An explicit physical core does not by itself guarantee the invariances tested by the suite” is strong and useful, but should distinguish more clearly between physical core, parameter transfer, and adapter choices .
- Formatting issues
- Tables and figure captions appear dense and, in some extracted segments, difficult to parse; verify final typesetting of Tables 4, 5, and 15 and of Figures 2, 4, and 5 for journal readability .
- The long author list and affiliation block on page 1 may benefit from cleaner vertical separation to avoid visual crowding in the preprint format .
- The notation for equations and units appears mostly consistent, but special characters and superscripts should be checked in the final production file, especially around mm d−1, W m−2, and tolerance notation such as 10−6 .
- AI content analysis for this post-2021 manuscript
Estimated percentage of AI-generated content: low to moderate, approximately 10–20%.
Basis for estimate:
- The manuscript has a highly regular sentence cadence, unusually even rhetorical structure, and repetitive parallel framing in several sections, which can be consistent with AI-assisted drafting.
- However, the work also contains specific domain equations, benchmark architecture details, artifact-governance discussion, model-by-model failure descriptions, and nuanced methodological caveats that strongly suggest substantial expert human authorship and editing .
Highlighted possibly AI-assisted sections:
- The introductory framing around benchmark motivation and the Turing analogy on pages 2–3 is polished, symmetrical, and stylistically generic in places, e.g., “The question changes from how close a model comes to 1 record to what the model must do under any record” .
- The limitations section on page 21 is systematically structured as “First… Second… Third… Fourth… Fifth…,” which may indicate AI-assisted organizational drafting, though this is also common in careful human scientific writing .
- The governance prose on pages 18–20 is notably template-like in sentence architecture .
Assessed epistemic impact:
- Even if portions were AI-assisted at the drafting level, I do not see evidence that such assistance undermines the manuscript’s core scientific contribution.
- The epistemic burden here lies in artifact reproducibility, benchmark validity, and correctness of reported verdicts—not in whether prose was partially machine-assisted.
- Therefore, the likely impact of any AI-assisted writing on the claimed contribution is low, provided the code, archives, and exact evaluation records reproduce the paper’s tables and figures.
Recommendations
-
Reframe the paper’s contribution more precisely as a controlled synthetic physics-benchmark for hydrologic model falsification and stress-testing, rather than as a broad validation framework for all hydrologic deployment contexts.
-
Add a dedicated subsection on score interpretability under missingness: explain how N/A rates affect cross-model and cross-class comparisons, report family-specific applicability, and discourage naive comparison of pass fractions across models with very different output declarations.
-
Make adapter dependence explicit in the main results section, not only in limitations: include a concise reporting standard for each evaluated model covering unsupported inputs, imputed attributes, transformed forcings, and known sensitivity points.
-
Strengthen the related-work section with a sharper novelty matrix: contrast HydroTuring against hydrograph metrics, physics-informed training constraints, and existing AI-for-science benchmarks along axes such as hidden test generation, law-specific pass/fail criteria, black-box compatibility, and reference-model validation.
-
Rebalance the discussion to emphasize the empirically strongest finding: the main practical value appears to be detection of perturbation, invariance, and state-bound failures rather than whole-record closure failures.
-
Add a short subsection on severity: explain how readers should interpret binary FAIL outcomes that arise from marginal versus major threshold exceedance, and whether future versions may include graded diagnostics.
-
Verify and clean all PDF formatting, page-break collisions, encoding artifacts, and spacing issues before journal submission.
-
Ensure the reproducibility package is audit-ready: because the manuscript’s credibility depends heavily on the benchmark archive, manifests, adapters, and seeds, those materials should be easy for reviewers to run and compare against the paper’s reported totals .
Overall recommendation: promising and potentially impactful methodological contribution, but it requires meaningful revision to calibrate claims, clarify interpretability, and reduce the risk of over-reading benchmark outcomes beyond the synthetic and adapter-mediated conditions actually tested.
Reviewer3 (AI review)
AI Review (reviewer3.com)
Overview
HydroTuring Benchmark Suite
Contribution
The authors introduce HydroTuring (v0.1.0), a black-box benchmark suite containing 37 physical probes designed to systematically test mass, energy, and momentum conservation in hydrologic models.
Runtime Black-Box Test Harness
Contribution
The framework evaluates containerized models using runtime-generated forcing and hidden tolerances, ensuring that models cannot memorize fixed test sets or tune outputs to pass physical criteria.
Multi-Model Physical Consistency Assessment
Contribution
Through evaluating 32 hydrologic models across learned, process-based, and reference architectures, the study demonstrates that standard hydrograph skill metrics conceal physical conservation failures, and that enforcing single-law architectural constraints does not guarantee physical consistency across other laws.
Comments (15)
1. Notation collision for symbol G in Equations 1 and 2 — Editorial
In Section 3.2.1, Equation (1) defines as "an exchange that the model declares with the outside" in the water budget (gwex, in ). Immediately following in Section 3.2.2, Equation (2) uses the identical symbol to denote "the ground heat flux" in the surface energy budget (hfg, in ) without noting the change in meaning.
2. Mismatch in whole-record residual reporting count on catchment-closure — Editorial
Among the 22 evaluated models that report a whole-record residual on catchment-closure , 21 are below 0.2% and the largest is 2.5% ( xaj
Several passages in the text and tables state that 22 evaluated models report a whole-record residual on catchment-closure. For example, Section 3.4.2 states: "Among the 22 evaluated models that report a whole-record residual on catchment-closure, 21 are below 0.2% and the largest is 2.5% (xaj)." (This 22-model count also appears in Section 4.2, Section 6.1, Table 13, and the legend of Figure 5a: catchment (22)). However, Figure 5a is titled "24 models · 3 closure probes" and plots a blue circle (catchment) for all 24 listed evaluated models, and Figure 5b explicitly states for closure (whole-record): "0 failed pairs (all 24 reporting models pass)". Furthermore, in Figure 4, 27 models are scored on catchment-closure (22 PASS and 5 FAIL). In Section 4.2, the 5 models that fail catchment-closure on another criterion (hype, raven, shud, simhyd, summa) are explicitly stated to "close their overall water budgets," and two of them (hype and raven) are even plotted in Figure 5a with catchment residual dots. Thus, the paper contradicts itself on whether 22, 24, or 27 evaluated models report a whole-record closure residual on catchment-closure.
3. Figure 5a omits one of the four closure probes — Editorial
a , Whole-record closure residuals that the evaluated models report on the closure probes (catchment, ungauged-basin and snowpack), as a share of cumulative precipitation, with 1 dot per model and probe, and the same quantity for the broken reference reference_leaky . The dashed line marks the 5% limit that the 4 closure probes share.
The caption to Figure 5a states: "Whole-record closure residuals that the evaluated models report on the closure probes (catchment, ungauged-basin and snowpack)... The dashed line marks the 5% limit that the 4 closure probes share," and the header of Figure 5a reads "24 models · 3 closure probes". However, Section 3.2.1 and Section 4.2 establish that there are 4 closure probes (catchment-closure, snowpack-mass-closure, extreme-event-closure, and ungauged-basin-closure), and Table 8 confirms that extreme-event-closure also scores the whole-record closure criterion (in addition to event_water_closure), which Section 4.2 directly discusses ("The model xaj passes the whole-record criterion with a residual of 2.5%"). Treating "the closure probes" as only 3 probes in Figure 5a while referring to "the 4 closure probes" in the next sentence of the caption and in Section 4.2 is internally inconsistent.
4. Notation collision for symbol A in Equations 1 and 3 — Editorial
where P is precipitation, E is evapotranspiration including sublimation, Q is runoff, G is an exchange that the model declares with the outside, A is a prescribed human water withdrawal, and ∆ S is the change in each reported store. The 24 mass probes fall into 5 families.
In Section 3.2.1, Equation (1) defines as "a prescribed human water withdrawal" (with units of water depth/flux). In Section 3.2.3, Equation (3) uses the identical symbol to denote "the flow area" of a channel cross-section without noting the redefinition.
5. Uncalibrated process models compared against real-basin trained AI models — Critical
We evaluated 32 models on the 37 probes of suite 0.1.0. Three are the physically exact references flex_ lumped , flex_topo and sacsma_snow17 . Seven are AI-based: an LSTM, the differentiable HBV model dhbv2 (Song et al., 2025), and 5 training checkpoints of a mass-conserving perceptron, mcp1 to mcp5 (Wang and Gupta, 2024). The remaining 22 are process-based models, from conceptual bucket models to
The benchmark does not specify how parameters were selected for the 22 process-based models on the invented 250 km² catchment, whereas the AI-based models (lstm, dhbv2, mcp1–mcp5) use weights and parameters fitted to real observed catchments. Because process-based model behaviors (such as recession constants, capacities, and evapotranspiration scaling) depend on parameterization, comparing uncalibrated or default-parameterized process models against pre-trained neural networks confounds model architecture with arbitrary parameter initialization and out-of-distribution shift.
6. Sub-daily time-step incompatibility scored as physical failure in dhbv2 — Major
and warming probes. However, it fails 6 probes in the invariance and response families. Its runoff differs by 12.1% of precipitation between hourly and daily records (limit 10%), likely because the calibrated parameters retain the daily time-step scaling at which they were fitted. Runoff also differs by
Similar to the LSTM case, dhbv2 is reported as failing resolution-invariance (12.1% difference vs 10% limit) because its calibrated parameters retain the daily time-step scaling at which they were fitted. Evaluating a daily-calibrated model at hourly and 1-minute resolutions without rescaling time-step-dependent parameters tests adapter implementation and time-step compatibility rather than an intrinsic failure of the differentiable physical core.
7. Exact reference models passed by construction via acceptance gate — Critical
exactly. The models flex_lumped and flex_topo are 2 conceptual models of the FLEX family (Li, 2026), and sacsma_snow17 couples the Sacramento soil moisture accounting model with the Snow-17 snow model (Upstream Tech, 2026). The 14 remaining exact references are written for the probes whose
The three physically exact reference models (flex_lumped, flex_topo, sacsma_snow17) are included in the 32 evaluated models and highlighted as the only models passing every applicable probe. However, these exact same models serve as the must_pass baselines in the acceptance gate (ht gate) across 24 to 25 probes (Table 11), meaning probes and their engineering tolerances were explicitly required to pass these models before being merged into the suite. Presenting their 100% pass rate as an empirical evaluation result is circular, as they are guaranteed to pass by the benchmark's construction.
8. Contract violation conflated with physical failure on human abstraction — Minor
Fourth, most models do not account for a prescribed human withdrawal. On mass/human-abstraction , 20 of 27 scored models show no change in any budget term when the probe withdraws 380 mm of water over 10 years. Two more account for part of it:
In Section E.1, the paper explicitly defines declaring a forcing input in model.yaml that the adapter does not actually consume as a contract/packaging error, and Table 4 defines INCOMPATIBLE (N/A) for models that do not support a required input. Yet on mass/human-abstraction, 20 models that show zero change in any budget term—indicating their adapters declared abstr in needs_forcing without passing it to the model or the model having an irrigation module—are scored as scientific FAIL verdicts rather than flagged during contract verification or marked INCOMPATIBLE, inflating the failure count for this criterion.
9. Unreported data and setup for synthetic calibration experiment — Major
In a synthetic calibration experiment, we fitted 2 conceptual models to the same daily runoff record. Model A ( reference_bucket ) conserves water, while model B ( reference_leaky ) sends 15% of its water input to a sink that its outputs never report. Against the observed streamflow, their Nash-Sutcliffe
The synthetic calibration experiment comparing reference_bucket and reference_leaky (yielding NSE values of 0.97 and 0.93) is central to motivating the benchmark, yet the source of the observed daily runoff record, its length, the calibration objective and algorithm, and the model parameterizations are entirely omitted from the text and appendices.
10. Prescribed static capacities vs internal model parameterization — Minor
on another criterion. The model hype lets soil storage reach 363 mm in a 320 mm profile. The models raven and shud evaporate 1.01 and 1.27 times the potential evaporation that the probe supplied. The model simhyd exceeds both its canopy capacity and the potential evaporation, and summa lets canopy storage reach 20.3 mm against a capacity of 2 mm. Although these models close their
The state_bounds and et_plausible failures on catchment-closure test whether reported storage and evaporation stay below the probe's static attributes (soil_capacity_mm = 320 mm, canopy_capacity_mm = 2 mm) and supplied pet. However, if a model's adapter does not map soil_capacity_mm or canopy_capacity_mm to overwrite the model's internal multi-layer porosity, root-zone depth, or leaf-area-based interception capacity (e.g., summa reaching 20.3 mm canopy storage or hype reaching 363 mm soil storage), the failure reflects a mismatch between the probe's nominal scalar attribute and the model's internal state definition rather than an unphysical conservation failure.
11. Pseudoreplication in AI-based model sample — Critical
. Seven are AI-based: an LSTM, the differentiable HBV model dhbv2 (Song et al., 2025), and 5 training checkpoints of a mass-conserving perceptron, mcp1 to mcp5 (Wang and Gupta, 2024). The remaining 22 are process-based models, from conceptual bucket models to
Five of the seven AI-based models (mcp1 to mcp5) are simply different training epoch checkpoints of the exact same mass-conserving perceptron architecture, contributing 101 of the 133 scored AI model-probe pairs (76%). Treating these five checkpoints as independent models heavily weights the aggregate 74% AI pass rate toward a single architecture and leaves only three distinct AI architectures evaluated (lstm, dhbv2, and mcp), which is too small and non-independent a sample to support general conclusions comparing AI-based and process-based model classes.
12. Lack of independent validation beyond calibration reference models — Major
A probe enters the suite only when its gate is satisfied. The gate has 2 kinds of check. A must-pass check requires an exact reference to pass the probe. A must-fail check requires a broken reference to fail the probe and to trip the criterion that it was built to trip. The 37 probes carry 116 must-pass and 78 must-fail checks. We re-ran all 37 gates on 2 November 2026 at the commit that tags suite 0.1.0, and every check
Each probe's engineering tolerances were chosen and gated using the 18 exact and 59 broken reference models, and many of the broken models were specifically constructed with large, fixed defects (such as a 15% sink or a 25 mm/d hard cap) for a single target probe. Showing that all 37 probes pass their acceptance gates demonstrates internal consistency with the design targets, but without testing against an independent set of defects or varying defect magnitudes beyond catchment-closure, it does not establish how well the probes discriminate subtler physical violations across the remaining 36 probes.
13. Adapter-induced failure on LSTM resolution invariance — Major
Hourly records produce a runoff volume that differs by 73.8% of precipitation from daily records of the same weather, because the network accepts arbitrary step sequences without an explicit time-step parameter and treats each hourly rate as a daily depth. Added rain also reduces subsequent daily runoff
The lstm failure on resolution-invariance (73.8% runoff difference between hourly and daily runs) occurs because the pre-trained network was trained exclusively on daily data and takes step sequences without a time-step input, so feeding it hourly forcing through the adapter causes it to interpret each hourly rate as a daily depth. Because the native model does not support sub-daily time steps, this model-probe pair should have been marked N/A (INCOMPATIBLE) by the manifest (timestep: [PT1D]) per Section 3.1 rather than configured by the adapter to run at unsupported sub-daily steps and scored as a physical failure.
14. Unadjusted pass rates across non-overlapping probe subsets — Major
probes, which says little because the sample is small. The 7 AI-based models pass 98 of 133 scored pairs (74%), while the 22 process-based models pass 302 of 398 (76%). Because these pass rates are nearly identical, model class alone does not predict physical consistency. Table 15 gives the standing of every
The aggregate pass rates for AI-based models (74%) and process-based models (76%) are computed by pooling all scored pairs within each class without accounting for which specific probes each model was evaluated on or clustering by model. Because different models are scored on different subsets of probes with widely varying difficulty (e.g., lstm is not scored on human-abstraction, where 81% of scored models fail, and AI models are barely scored on momentum or energy probes), comparing raw pooled pass fractions across classes is confounded by probe selection.
15. Adapter omissions conflated with native model limitations — Major
FAIL and 579 are N/A. Among the N/A pairs, 499 are INCOMPLETE and 80 are INCOMPATIBLE. Consequently, 49% of all model-probe pairs remain unscored due to missing output variables or time-step mismatches rather than physical failures. Mass probes are N/A for 30% of pairs (234 of 768), while
The INCOMPLETE verdict reflects variables omitted from the model.yaml manifest and ht_adapter.py wrapper rather than variables the underlying model is genuinely incapable of computing. For instance, comprehensive land surface and distributed hydrologic models such as summa, vic, noahmp, mhm, and wflow_sbm natively compute snow water equivalent and energy or routing states, yet are marked INCOMPLETE on snowpack-mass-closure and several energy probes (Figure 4), confounding incomplete adapter implementations with intrinsic model omissions.
XSci (AI review)
Summary
The authors argue that hydrograph skill cannot establish whether a hydrologic model respects conservation laws, illustrating this with two calibrated models whose Nash-Sutcliffe efficiencies are 0.97 and 0.93 although the second loses 15% of its water input to an unreported sink. They introduce HydroTuring v0.1.0, 37 black-box probes (24 mass, 7 energy, 6 momentum), each a controlled experiment on forcing generated at run time from a recorded seed, with criteria and tolerances held on the host while the model runs sealed in a container. Applied to 32 models, the suite returns 474 PASS, 131 FAIL and 579 N/A over 1,184 pairs; a pre-trained flood-forecasting LSTM fails 6 of its 11 scored probes, and 49% of pairs cannot be scored because models omit the required variables.
Contribution and Significance
The core contribution is an instrument rather than a result: a conservation-law test suite that runs on a delivered model without access to its weights, source or training data, on cases generated at run time that cannot have been seen. The hidden contract, the paired control-and-perturbation cases that catch a model ignoring an input, the labelled regimes that stop errors of opposite sign cancelling, and above all the acceptance gate are the substance of the paper. The gate is a standard of self-validation rare in Earth system science: a probe is admitted only if declared exact references pass and declared broken references fail the criterion each was built to trip (Tables 11 and 12).
The novelty relative to the closest prior work is real, and correctly argued. PDEBench (Takamoto et al., 2022) reports deviation from conserved quantities, but on fixed public datasets for PDE surrogates; CheckList (Ribeiro et al., 2020) supplies the behavioural-testing template without the physics; MC-LSTM (Hoedt et al., 2021), the mass-conserving perceptron (Wang and Gupta, 2024) and hard-constrained networks (Beucler et al., 2021) build physics into training and never test the delivered model. The empirical half of the paper is much weaker than the instrument, and its headline numbers establish less than claimed.
Strengths
- The gate is documented rather than asserted: Table 12 names the defect, probe and criterion for each of the 59 broken references, and all 37 gates were re-run at the tagged commit.
- The seed analysis of Section 3.4.2 is rarely made and is right: reference_saturating would pass a single-seed extreme-rain probe 40% of the time, and 116 of 120 three-seed subsets expose it.
- Section 4.4 holds the most interesting finding: all five mass-conserving perceptron checkpoints close the budget below 0.001% of precipitation yet fail 2 to 6 other probes, mcp1 (epoch 27) failing 2 and mcp4 (epoch 2707) failing 6.
- Section 6.3 is candid, admitting that one seasonality attribute changed the runoff coefficient of variation of dhbv2 from 1.1 to 0.07 and flipped two verdicts.
Recommendation
Major Revision for Water Resources Research. The instrument is novel, carefully validated and of clear use to this readership, and every repair below lies inside the manuscript's own approach: re-framing the lead result, auditing FAIL against N/A, an adapter sensitivity analysis, and restraint in the energy and momentum claims. What decides it is that the three most quotable results are each weaker than stated, while the two that survive scrutiny, the checkpoint variability and the 49% unscorable rate, are buried.
Reject for the International Conference on Machine Learning: the manuscript proposes no learning algorithm, architecture or theory, and only 7 of 32 models are learned, so for that venue the advance is too far outside its scope.
Reject for Nature Communications: the results that would carry the broad interest that venue requires, the class comparison and the LSTM study, are exactly those the evidence does not support.
The concerns below are written against Water Resources Research.
Major Concerns
-
The lead result is guaranteed by the benchmark's own design (Abstract; Sections 4.1 and 7; Tables 11 and 15). The abstract and conclusion report that "only the 3 exact references pass every probe among the 28 models scored on 5 or more probes". Those three references are the gate's must-pass baselines: Table 11 records that 25 probes require flex_lumped to pass, 25 require sacsma_snow17 and 24 require flex_topo, and Table 15 reports 25/25, 25/25 and 24/24. The agreement is exact, and necessarily so: a probe failing these models could not have been merged. This is a consistency check on the suite, not a finding about models. Either evaluate exact models never used in any gate, or call it a design guarantee and drop it from the abstract.
-
Verdicts confound model physics with adapter packaging (Sections 3.1(1), 4.2, 4.3, 5.1, E.1). Design principle (1) states that "if a model does not support sub-daily inputs, the resolution_invariance criterion would be marked N/A". Section 4.3 then explains the LSTM's 73.8% failure by saying the network "accepts arbitrary step sequences without an explicit time-step parameter and treats each hourly rate as a daily depth" — a model that does not support sub-daily input — yet it is scored FAIL and counted in the abstract's "fails 6 of the 11". Separately, human_abstraction is the most frequent failing criterion (22 of 131 failed pairs), and 20 of 27 scored models show no change in any budget term when 380 mm is withdrawn; since Section E.1 names "declaring a forcing input that the adapter does not consume" as a common failure mode, those verdicts are as consistent with declaration errors as with physics — against Section 5.1's claim that "a failed probe is a scientific finding rather than a packaging defect".
Adapter choices are already shown to decide verdicts: one attribute flipped two dhbv2 verdicts, and the warming-response failure, 0.088 against a limit of 0.10 on 3 of 5 seeds, strengthens when the static attributes update. Please audit which FAIL verdicts should be N/A under the manuscript's own rules, and add a sensitivity analysis.
-
The class comparison cannot be made from these data (Abstract; Sections 4.1 and 6.1; Table 15). That the two classes "pass similar shares of their scored probes (76% and 74%)", and therefore that "model class alone does not predict physical consistency", rests on 302/398 and 98/133. The fractions come from different probe sets, since N/A varies by model: noahmp is scored on 25 probes, mcp1 on 20 with no momentum probe. Five of the seven AI models are checkpoints of one architecture (Section E.3 gives epochs 27, 3, 619, 2707 and 2999), so that sample holds three independent models, not seven. No uncertainty is attached, and removing the LSTM gives 93/122, or 76.2%, indistinguishable from the process-based share. Either restrict the comparison to probes both classes are scored on and treat the checkpoints as one model, or drop the inference.
-
Energy and momentum are in the title but barely exercised (Abstract; Sections 4.1 and 6.3; Table 15). Of 224 energy pairs 188 are N/A, and snowmelt-energy-water and surface-energy-closure have scored no model at all; of 192 momentum pairs 157 are N/A, and most momentum entries in Table 15 are 1/1 or 1/2. Section 6.3 adds that only three models are scored on the two daily cross-budget probes, that all three fail, and that those scores are "provisional" because of a known radiation mismatch between probe and adapter. The suite is therefore demonstrated for mass and a prototype for the other laws; the title, abstract and Section 7 should say so, and the provisional verdicts be marked in Figure 4 and Table 15.
Minor Comments
- Section 4.4 reports 10,040 mm of runoff against 8,686 mm of rain in the dhbv2 precipitation-counterfactual control run, although Table 8 lists closure among that probe's criteria and Sections 4.2 and 6.1 conclude that closure holds.
- The abstract and Section 7 drop the Section 4.3 qualification that the evaluation reflects released weights with this adapter "rather than the full operational forecasting system".
- In Section 4.2 the event count differs by model on one probe (xaj 1,316, shud 1,257, cwatm 1,257, wflow_sbm 1,299) although the forcing is seeded; please define an event.
- The motivating experiment of Section 1 is never described: give the record, the period, the calibration algorithm and the objective function used.
- Section 4.4 should engage Frame, J. M., Kratzert, F., Gupta, H. V., Ullrich, P., and Nearing, G. S., "On strictly enforced mass conservation constraints for modeling the rainfall-runoff process", Hydrological Processes 37(3):e14847, 2023.
- Section 4.3 should cite Gauch, M., Kratzert, F., Klotz, D., Nearing, G., Lin, J., and Hochreiter, S., "Rainfall-runoff prediction at multiple timescales with a single Long Short-Term Memory network", HESS 25:2045-2062, 2021.
- Section 3.4.1 calls reference_streamflow_only "not broken but limited" although Table 12 counts it among the 59 broken references.
- Table 7, the authoritative lexicon, is misaligned in the version received, giving snm the meaning "Latent heat flux".
- No runtime or computational cost is reported, which a reader needs to judge the claim that the suite is a practical smoke test.
- Since Section 5.2 records that probes merged before 16 September 2026 were accepted by the maintainer alone, state how many of the 37 had two reviewers.