River early warning across 671 British catchments
- CAMELS-GB v2
- Forecasting
- Unseen regions
- Uncertainty
- In progress
Setup
- The question behind the design: how far ahead can river flow and flood or low-flow warnings be forecast without knowing future weather, and does that skill carry over to places the model was not trained on?
- Daily river flow and weather for 671 catchments, 1970 to 2022. 659 had enough data to use.
- Task: on day t, forecast flow 1, 3 and 7 days ahead, using only what is known on day t. No future rainfall.
- Train on 1970 to 2004, choose settings on 2004 to 2012, test on 2012 to 2022.
- Warnings: flow above a catchment's 95th percentile (high) or below its 10th (low), with thresholds taken from the training years.
Results
Test years, median across catchments. NSE is 1 for a perfect forecast and 0 for a forecast no better than the long-term average.
| Model | 1 day | 3 days | 7 days |
|---|---|---|---|
| "Same as today" (persistence) | 0.509 | 0.066 | -0.235 |
| Gradient boosting, one model per catchment | 0.761 | 0.380 | 0.242 |
| Upper bound: the same model given the weather that happened | 0.865 | 0.810 | 0.722 |
- Most of the skill beyond a day depends on knowing the weather. At 7 days the model reaches 0.24 without future weather and 0.72 with it.
- A whole unseen region costs little. One regional model, tested on regions held out of training, loses a median of 0.019 NSE at 1 day.
- A different climate costs much more. Trained on wet catchments and tested on dry ones, NSE falls from 0.850 to 0.752. A random-half control loses about 0.03.
- A single-number forecast fails as a warning system at 7 days (high-flow critical success index 0.005). Warnings taken from forecast quantiles reach 0.175. A model-free band around today's flow reaches 0.155, so the model's edge is narrow.
- Most multi-day high-flow warnings are false alarms. For the quantile model 75 to 80% are wrong. For the two persistence baselines the share runs from 64% to 82%.
- The model's 90% range is honest where it was trained (coverage 0.90 to 0.91 after conformal calibration) and less so elsewhere: on unseen regions about one catchment in six gets a range that covers under 80%.
What went wrong, and what changed
- The first diagnosis of a weak baseline (early stopping) was wrong. A controlled comparison on validation years showed the target scale was the cause. The log records both.
- Twelve of 51 quantile models hit the tree limit, so those are under-trained. Reported, not hidden.
Limits
- "Unseen" is not "ungauged": recent flow at the catchment is still an input.
- The uncertainty experiment used one random seed. Coverage during floods specifically has not been checked.
- Still to do: a neural sequence model, gaps in the inputs, and explanations.
Code, protocols and decision log: repository to be published.
Drug response beyond a single marker
- GDSC and Cell Model Passports
- Leakage
- Missing data types
- Cluster stability
- Explanation stability
- Label noise
Setup
- 924 cancer cell lines. One gene, PTEN, is a known but incomplete guide to how they respond to drugs that block the AKT protein.
- The question behind the design: how much more of the response does wider molecular data explain, and can the patterns be trusted?
- 5-fold cross-validation repeated 5 times on saved folds. Every step that learns from data happens inside the training folds.
- Cell lines are a stand-in. Nothing here is about patients or treatment.
Results
All numbers are held-out. R² is the share of the variation in response that a model explains.
| Finding | Number |
|---|---|
| PTEN status alone (R²) | 0.05 |
| PTEN + cancer type + sequencing centre (R²) | 0.11 |
| Adding 2,000 genes: elastic net, then gradient boosting (R²) | 0.14, 0.19 |
| Choosing genes before cross-validating, on shuffled data with nothing to find (R²) | 0.14, all false |
| Adding protein data on top of gene expression | no gain |
| Overlap between the top-20 gene lists of refitted models | 0.20 |
| Agreement between two measurements of a related drug | 0.81 |
- Mostly a negative result. Wider data helps a little. More than four fifths of the response stays unexplained.
- That gap is not mainly measurement noise. Where a related drug was measured twice, about 80% of the variation repeats, and the best model explains 18%.
- A common mistake produces a believable result from noise. The false score is as large as the real one.
- The only stable subgroups are tissue of origin. Made-up data with no groups at all scores 0.80 to 0.87 on the usual stability measure when cut in two, so stability alone proves little.
- Explanations change when the model is refitted. Only the original marker and one gene are reliably in the top 20.
What went wrong, and what changed
- Gains were first computed on predictions averaged over repeats, which flattered the less stable model (+0.094 became +0.076). Corrected, with a test.
- A sanity check declared in advance turned out nearly empty: 10 of the 12 pathway genes it named had been filtered out before the models saw them.
- The pre-set bar for "stable clusters" had no reference to compare with. A no-groups reference was added before the run.
- Three analyses were added after seeing results. They are labelled exploratory in the protocol and wherever they are reported.
Limits
- One drug screen from one laboratory. No independent dataset.
- Missing protein data was simulated at random. Models were not tuned.
- No biological interpretation has been checked by a biologist.
Code, protocol and decision log: repository to be published.
How the studies are run
- Protocol firstThe design is written and dated before any model is fitted. Later changes are dated amendments.
- Baselines firstEvery model is compared with the simplest thing that could work, on exactly the same held-out data.
- Fixed splitsSaved once with a fingerprint. The code refuses to overwrite them.
- Nothing learned from held-out dataFeature choice, scaling, gap filling and tuning all happen inside the training data.
- Negative results reportedCriteria for "this did not work" are set in advance and reported against.
- A log of every decisionIncluding the mistakes, how they were found and what was changed.
How AI tools were used
The protocols, code, tests, runs and first-draft documentation for both studies were produced with an AI assistant (Claude), working under research rules recorded in each repository. Every choice it made on my behalf is listed in that repository's LOG.md, for me to confirm or change.
No prediction was written down before the results existed, so none is claimed here. The questions, the summaries on this page and the reports in each repository were also drafted by the assistant from the logged results.
Next
- A neural sequence model for the river study, and its behaviour with gaps in the inputs.
- A third study on satellite flood mapping with imperfect labels.