Methodology

What each check asks

Walk-forward

Re-fits the strategy on an earlier window and tests it on the next, unseen one — asking whether the edge survives on data it was never tuned on.

Parameter sensitivity

Nudges each parameter up and down — asking whether performance sits on a broad plateau or balances on a single lucky value.

Deflated Sharpe

Discounts the Sharpe ratio for how many parameter combinations were tried — asking whether the best result is better than you'd expect from luck alone across that many tries.

Regime breakdown

Splits returns by market condition — asking whether the edge is broad, or just one bull run wearing a strategy costume.

What each verdict means

PASS

The edge held up across every check we could run: it survived out-of-sample periods, small parameter changes, the multiple-testing penalty, and different market regimes. Encouraging — not a guarantee.

SHAKY

There is a signal, but at least one check flagged fragility — performance leaned on a narrow parameter range, one regime, or thinner out-of-sample results. Worth a closer look before trusting it.

LIKELY OVERFIT

The strong in-sample result mostly evaporated where it counts — out of sample, under small parameter changes, or after the multiple-testing penalty. A good-looking curve that probably won't repeat.

UNTESTABLE

There wasn't enough evidence to judge — too few trades, too little history, or windows too short to validate. Not a pass and not a fail: we won't pretend to know what the data can't tell us.

Why a great-looking curve can still be overfit

Try enough parameter combinations on one slice of history and some will look brilliant by chance alone — the equity curve fits the noise, not a repeatable edge. These checks exist to separate a real signal from a flattering coincidence, which is exactly what a single backtest number can hide.

Key terms

In-sample vs. out-of-sample

In-sample is the data a strategy was tuned on; out-of-sample is data it never saw. A strategy that performs well in-sample and poorly out-of-sample was memorized, not discovered. Nearly every overfit strategy looks strong in-sample — that is precisely the problem.

Overfitting

Fitting the noise of one historical period instead of a real, repeatable pattern. The signature is fragility: small changes to the parameters or the time window cause the returns to collapse.

Walk-forward validation

Rather than testing on all history at once, the data is divided into sequential segments: the strategy is tuned on one, tested on the next, then rolled forward and repeated. It approximates running the strategy through time instead of grading it with hindsight. A large gap between tuned and walk-forward performance indicates the edge does not travel.

Deflated Sharpe ratio

A conventional Sharpe ratio becomes more flattering the more variants are tried, because with enough attempts something scores well by chance. The Deflated Sharpe ratio corrects for how many effective variations could have been tested, then asks whether the result remains convincing once that search is accounted for.

Regime concentration

Whether returns were earned across varied market conditions or accumulated during a single favorable period. A strategy that produced its gains in one bull run, and little elsewhere, is not robust — it is a wager on that condition recurring.

Parameter sensitivity

How sharply performance depends on exact parameter values. A robust edge occupies a broad plateau: adjusting a threshold barely moves returns. A fragile one sits on a narrow peak, where the tuned values look strong and everything around them falls away — a hallmark of curve-fitting.

Verdict (Pass / Shaky / Likely overfit / Untestable)

Deflate's single summary judgment after all checks. Untestable does not indicate a poor strategy; it means the data or structure did not permit an honest test, which Deflate reports rather than disguising as a grade.