A CAMELS-531 Hydrology MIP — Coming Soon
Standardized structural and end-to-end benchmarking across 3 Spatial Regimes (Gauged Simulation, 12-Fold Random PUB, and 18-Fold HUC-02 PUR) × 4 Forcing Categories (Single CAMELS, Multiple CAMELS, With Lagged Streamflow, and Freeform Forcings). Every leaderboard entry links directly to (i) its Data Archive (Zenodo / HydroShare) and (ii) its Open-Source Codebase (GitHub).
Community Contributions to the CAMELS-US MIP
Submit new process-based, conceptual, hybrid, or machine learning models to any of the 12 leaderboards by providing (i) a permanent, DOI-minted 531-basin daily hydrograph archive and (ii) an open-source codebase.
§3.1 How to Contribute & Mandatory Repository Requirements
The 12 leaderboards in Section 4 are designed as a living community benchmark. Modeling groups can submit new entries to any of the 12 leaderboards (Gauged, 12-fold random PUB, and 18-fold HUC-02 PUR across the four forcing categories) either directly through the automated scoring & verification workbench on the right or by opening a pull request against the CAMELS-US MIP repository.
Hosted on Zenodo, HydroShare, or an equivalent open-access repository containing daily simulated streamflow across all 531 benchmark basins for the 1 Oct 1989 – 30 Sep 1999 Newman evaluation decade (water years 1990–1999; 3,652 daily steps).
Hosted on GitHub or an equivalent version-controlled platform containing the model implementation, configuration files, and exact training or calibration scripts needed to reproduce the archived hydrographs.
Four Rules of Responsible Benchmarking Practice (§3.1)
Rule 1 Strictly Blind Water Years 1990–1999 Evaluation Decade
The 1989-10-01 to 1999-09-30 Newman evaluation decade (water years 1990–1999) must be treated as a strictly blind holdout: model architectures, hyperparameters, loss functions, and early-stopping criteria must be selected using only data from the water years 2000–2008 training and calibration period, and never by inspecting test-decade performance.
Rule 2 Strict Spatial Holdout & Canonical Fold Assignments (PUB & PUR)
In the ungauged leaderboards (12-fold PUB and 18-fold HUC-02 PUR), no streamflow observations, calibrated parameters, or target-basin statistics from the held-out test folds may be used during training, normalization, or model selection, and all folds must be generated using the canonical fold-assignment lists provided in the MIP repository.
Rule 3 Honest Input & Forcing Category Placement
Models must be placed in the leaderboard corresponding to their actual inputs: any model that uses multiple CAMELS forcing products, external non-CAMELS meteorological datasets, or lagged observed streamflow (autoregression or data assimilation) must be submitted to the Multi-CAMELS, Freeform Forcings, or With Lagged Streamflow leaderboard rather than compared against single-forcing baselines.
Rule 4 Deterministic Daily Streamflow Forecasts
All submissions must consist of deterministic daily streamflow forecasts: contributors who use multi-seed ensembles, multi-calibration ensembles, or probabilistic models are responsible for reducing their predictions to a single daily hydrograph per basin prior to submission. Single-model runs and multi-model or multi-seed ensembles are listed as separate rows on the leaderboards.
Automated Verification, Live Scoring & Leaderboard Submission
Validate basin coverage (531 basins), date alignment (1989-10-01 to 1999-09-30, water years 1990–1999; 3,652 days), and repository links before computing the standardized metrics.
Table 2. Registry of Models Participating in the CAMELS-531 MIP
Registry of participating model families across Source Categories (i), (ii), (iii*), and (iv), with direct buttons to (i) the Data Archive and (ii) the Open-Source Codebase.
Section 2. The CAMELS-531 Model Intercomparison Protocol & Metrics
Unified experimental design anchored to the 531-basin temporal and spatial partitions across water years 1990–2008.
1. Target Catchment Set & Newman Period (§2.1)
- 531 Benchmark Basins: Curated from the 671 CAMELS-US catchments by excluding basins >2,000 km² or with >10% area discrepancy.
- Training / Calibration Window:
1999-10-01through2008-09-30(9 water years, water years 2000–2008; 3,288 daily steps per basin). - Blind Evaluation Window:
1989-10-01through1999-09-30(10 water years, water years 1990–1999; 3,652 daily steps per basin). - 100% Data Completeness: Across water years 1990–2008, 100% (531 of 531) of the benchmark basins have complete daily streamflow records and all three meteorological forcings.
2. Four Source Categories (§2.2)
- Category (i) — Literature: Models already trained/calibrated on the Newman period in peer-reviewed studies or public Zenodo/HydroShare archives.
- Category (ii) — MIP Re-Run: Models trained, calibrated, or re-evaluated from scratch across all 531 basins on the Newman period in this MIP.
- Category (iii*) — Non-Newman Archive (*): Large distributed models and multi-structure suites evaluated on alternative windows, marked with
*. - Category (iv) — Community Contributions: Models contributed via the Section 3 protocol with open-source code on GitHub and 531-basin daily hydrographs on Zenodo/HydroShare.
3. Standardized Performance Metrics (§2.1)
- Median NSE (Primary Ranking Metric): Basin-median Nash–Sutcliffe Efficiency across the 531 catchments over water years 1990–1999.
- Arithmetic Mean NSE: Unweighted mean NSE across all 531 basins (sensitive to severe outlier failures).
- Median KGE: Basin-median Kling–Gupta Efficiency balancing correlation, variability ratio, and mean bias.
- Median |FHV| (%): Basin-median absolute top 2% peak flow volume bias.
- Failure Basin Count (NSE ≤ 0): Number of basins out of 531 where the model performs no better than the basin's long-term mean flow.
4. Generating a Compliant 531×3652 Submission File
import numpy as np
# qsim: deterministic daily predictions (531, 3652)
# Window: 1989-10-01 to 1999-09-30 (water years 1990-1999, mm/day)
np.savez_compressed(
"model_camels531_newman_1989_1999.npz",
qsim=qsim_531x3652.astype(np.float32),
basins=basin_ids_531,
)
Community Comments & Technical Discussion
Organized, topic-scoped technical discussion and peer review across the 12 leaderboards, experimental protocol, forcing datasets, and model contributions.
1. Organized 5 Topic Channels & Table Scoping
Every comment is categorized into one of five structured technical channels and pinned to a specific scope (General, Table 2 Registry, or Leaderboard Tables 3–14) so methodological questions and leaderboard audits never get lost in an unstructured feed.
2. Accountable ORCID iD & GitHub Researcher Attribution
Contributors provide their name, institution, and optional ORCID iD (0000-000X-XXXX-XXXX) or GitHub handle (@username) to display a verified researcher badge and maintain scientific accountability.
3. Protected Pre-Publication Moderation & Anti-Spam
Submissions pass through automated bot honeypots, link-spam filters, and strict HTML sanitization, and enter a moderator review queue before appearing on the public feed (with optional 1-click export to GitHub Discussions).
Post a Structured Community Comment
Comments are organized by topic channel and reviewed by benchmark moderators prior to public listing.