The 395,186-race validation
The engine's weights were frozen, then run blind across 395,186 historic UK & Ireland races (1990–2020) — nothing was fit to this data, and no result was visible to the ranker. This page is that result, including the mistake an earlier version of this test caught in itself.
Baselines and what this does not prove
- The random baseline here understates itself. "Random ≈ 9.6%" was computed as 1/(average field size). The correct figure is the mean of 1/(field size) taken per race, and it is always the larger of the two when field sizes vary — so the real random baseline is above 9.6% and the engine's edge over it is smaller than the two numbers side by side suggest. The live page uses the corrected formula; recomputing it over this archive is a named follow-up, so it isn't quietly restated here as though it had been.
- This isn't yet compared to just backing the favourite. Market odds were this dataset's largest-coverage input, so the harder, more honest comparison is the engine against the crowd's own favourite — not against a random pick. That number is not computed here yet; it's the next measurement, not a claim this page is making.
- The archive's odds are the starting price — the price known only once the race goes off, not a price you could have taken in advance. ROI figures below use it because it's what the archive records, and that limit applies to every ROI number on this page.
- This archive is not today's live free-tier engine. The live UK/IRE feed on the free plan carries no odds field at all, so the market factor (the largest weight in the formula) has 0% coverage there and the engine silently runs a weaker fallback formula instead of the market-anchored one this backtest used. See the live track record for what's actually running today.
The headline numbers
Mean reciprocal rank 0.538, mean Spearman 0.406. A negative ROI at the starting price is expected — it's roughly the bookmaker's overround, not a sign the ranking is wrong (see the calibration table below).
Calibration — predicted vs. actual
Split every runner by the engine's predicted win probability; if the model is honest, the actual win rate in each bucket should track the predicted one.
| Predicted | Predicted % | Actual % | n |
|---|---|---|---|
| 0–20% | 6.0% | 6.2% | 3,564,765 |
| 20–40% | 27.3% | 26.5% | 418,370 |
| 40–60% | 48.3% | 45.1% | 94,439 |
| 60–80% | 67.3% | 63.9% | 26,767 |
| 80–100% | 84.9% | 84.0% | 2,974 |
By era — does it hold across decades?
| Era | Races | Win | Top-3 | Brier | ROI |
|---|---|---|---|---|---|
| 1990-1994 | 43,525 | 35.4% | 68.0% | 0.0887 | -7.4% |
| 1995-1999 | 46,051 | 34.6% | 67.4% | 0.0882 | -6.3% |
| 2000-2004 | 61,011 | 33.0% | 64.2% | 0.0801 | -9.7% |
| 2005-2009 | 63,589 | 32.2% | 64.1% | 0.0824 | -8.3% |
| 2010-2014 | 81,810 | 33.0% | 65.6% | 0.0904 | -9.8% |
| 2015-2019 | 84,482 | 33.6% | 66.5% | 0.0909 | -9.2% |
| 2020-2024 | 14,718 | 32.9% | 66.0% | 0.0871 | -11.0% |
Eras are five-year buckets named by their first year, so the last row holds 2020 alone — the archive ends there, and 2020 was a COVID-shortened season, which is why it carries fewer races than a full bucket. Its 14,718 races are already counted inside the 395,186 headline total above.
By field size
| Field | Races | Win | Top-3 | ROI |
|---|---|---|---|---|
| ≤6 runners | 70,267 | 45.5% | 83.2% | -6.3% |
| 7–12 runners | 214,113 | 33.1% | 66.1% | -9.0% |
| >12 runners | 110,806 | 26.4% | 54.3% | -10.1% |
Smaller fields are easier to pick (fewer ways to be wrong) and that shows up directly in the win rate — this is expected, not a sign the engine "prefers" small fields.
The mistake we caught
This archive's only speed-like figures (RPR/TR) are ratings assigned after the race — using them as a pre-race speed figure leaks the result into the input. An earlier version of this backtest did exactly that, then tuned weights on top of the leak. Here's what it looked like, measured on the same 2,905-race slice (stride sample, k=11, of 32,101 races across 2019–2020), scored three ways:
| Path | Win rate | ROI |
|---|---|---|
| Clean (default loader), DEFAULT_WEIGHTS | 32.7% | -9.9% |
| Leaky (leakyRatingsAsSpeed:true), DEFAULT_WEIGHTS | 35.7% | +2.9% |
| Leaky, in-sample tuned weights | 50.6% | +135.1% |
A +135.1% ROI is not a betting edge — it's the model reading its own answer key. The loader (src/lib/history-csv.ts) no longer maps those post-race ratings into a speed figure by default; every number above this section on this page is from the leak-free path.
Reproduce it
This backtest runs against the repo's own history archive, not a hosted service — anyone with the repo can re-run it:
npm run backtest:historyThis is a historic backtest, not the live app. For how the engine is actually performing right now — including where it's degraded to a weaker fallback formula — see the live track record →