Why most backtests are measuring their own assumptions

Most backtests are not measuring an edge. They are measuring the assumptions of the person who wrote them, reflected back with a decimal point.

This is not a claim that backtesting is useless. It is a claim that a result you cannot decompose is not evidence, and that the decomposition is the work.

Three ways a result inflates itself

Each of these is individually well known. What makes them dangerous is that they compound quietly, and none of them announce themselves in the equity curve.

Look-ahead

The most common form is subtle: a bar's close used to make a decision that would have been taken during that bar. If your rule references the session high, and you evaluate it on the bar that set the high, you have used information you did not have.

A useful discipline is to write every rule as a question you could have answered out loud, at the time, without knowing what happened next. If you cannot phrase it that way, the rule is not implementable.

Survivorship

Continuous futures contracts are stitched together from individual expiries. The stitching is a choice, and different choices produce different histories. A strategy that trades the roll window is often trading the artefact rather than the market.

We handle this by excluding the roll window from some engines entirely and by vetoing the session after expiry in others. Neither is obviously correct. What matters is that the choice is explicit and its effect is measured.

Selection

If you test two hundred parameter combinations and report the best one, you have not found an edge. You have found the maximum of two hundred draws from a distribution, and the expected maximum of two hundred draws from noise is comfortably positive.

The defence is not to stop searching. It is to hold out data you never touch, and to accept that the held-out result is the result.

A worked example

Consider a rule that buys the first pullback after a session opens above the prior day's value area, exits at the close, and is evaluated over five years of one-minute data.

Run naively, this shows a positive expectancy. Then:

• Correct the look-ahead in the pullback definition, and roughly a third of the edge disappears.
• Exclude the roll windows, and the remainder halves.
• Split the sample and check the second half alone, and what is left is not distinguishable from zero at any sample size you actually have.

None of these corrections were sophisticated. They were bookkeeping. The original result was not a discovery; it was three bookkeeping errors wearing a trenchcoat.

What a defensible result looks like

A result you can defend is one where you can state, in advance, what would have falsified it - and then show that it survived.

In practice that means:

• The rule is stated before the test, not fitted after it.
• The out-of-sample half is genuinely untouched.
• The result replicates on a related instrument without retuning.
• The failure modes are described as clearly as the successes.

That is a much shorter list of surviving ideas than most people expect. It is also the only list worth trading.