Backtesting basics: walk-forward testing and traps
A backtest replays a set of rules over past data and reports what would have happened. It is a useful way to reject ideas and a poor way to prove them. A backtest cannot know the future, and it is very easy to build one that looks better than the idea deserves. This guide covers the traps and the habits that guard against them. It is education about method, not a view on any strategy.
Key takeaways
- Judge a rule on data it never saw while you were designing it.
- The more variations you try, the more likely the best-looking one is luck.
- Costs, filing delays and dead companies are the usual sources of flattering results.
- A pass on past data is evidence about the past, not a promise about the future.
The traps
- Look-ahead bias. Using information that was not available at the time. A common version: treating a company's results as known on the period end date, when they were filed weeks later. A 10-Q is due 40 or 45 days after the quarter end and a 10-K 60, 75 or 90 days after the fiscal year end, depending on filer status (Form 10-Q instructions, Form 10-K instructions).
- Survivorship bias. Testing only on companies that still exist today, which leaves out the ones that failed. A bibliography the Library of Congress prepared for the SEC (August 2010) defines it as the disappearance of failed companies from performance indexes, so that they present misleading results (Library of Congress bibliography for the SEC).
- Overfitting. Trying many variations and keeping the one that fit history best. Researchers have shown that, given any financial series, it is relatively simple to overfit a strategy so that it performs well on the data used to design it (Bailey, Borwein, Lopez de Prado and Zhu, Notices of the AMS, 2014). The more variations tried, the more likely the winner is luck.
- Ignoring costs. Commissions, the bid-ask spread and slippage. A strategy that trades often can be wiped out by costs that look small per trade.
- Too few trades. A result from a dozen trades is mostly noise.
In-sample and out-of-sample
In-sample data is what you used to design or tune the rules. Out-of-sample data is history the rules never saw. The AMS paper above uses the same split and treats a backtest as realistic when in-sample and out-of-sample performance are consistent. A strategy that works in-sample and fails out-of-sample was probably fitted to noise. The honest test is always on the part you did not look at while designing.
Walk-forward testing
Walk-forward testing makes that separation repeat. Fit on one window, test on the window right after it, slide forward, and do it again. Each fold gives a separate out-of-sample result, so you can see whether performance is steady across periods or came from one lucky stretch. It is a common research method, not a regulated standard, so describe the window lengths when you report a result.
Baselines
A result means little without something to compare against. Two useful baselines are simply holding the asset, and entering at random times with the same exits. If the rules cannot beat random entries over the same folds, the entry rule is adding nothing. A strategy can also look good only because the market went up, which buy-and-hold makes visible.
Questions to ask of any backtest
- How many trades were there, and over how many separate periods?
- Were costs and slippage included, and at what level?
- Which data was used for design and which was held out?
- How many variations were tried before this one?
- Was every input available on the date the rule used it?
- What was the worst drawdown, and could you have lived through it?
- How does it compare with holding and with random entries?
A made-up illustration
Imagine a rule that scores well on a ten-year chart with a smooth upward curve. Split into five folds, two folds are flat, one loses money and two make all the gain. The overall curve hid that. The folds do not prove the rule is bad, but they show the single curve was telling a simpler story than the data. These numbers are invented to explain the idea.
Costs, in plain arithmetic
Suppose a rule trades 100 times a year on positions of 10,000, and each round trip costs 10 once spread, commission and slippage are counted. That is 1,000 a year in costs, before any view on direction. The same rule at 10 trades a year spends 100. This is why a high-turnover idea has to clear a much higher bar than a slow one, and why a backtest run with zero costs tells you very little about anything that trades often. The amounts are chosen to make the arithmetic easy and are not estimates of any real market's costs.
How much data is enough
There is no magic number, but the direction is clear. More independent trades across more distinct market conditions give a result you can lean on a little; a few trades in one trending year give one you cannot. Consecutive trades from the same trend are not independent. If your whole sample sits in one regime, say so when you describe the result, and treat it as a statement about that regime only.
What a pass means
Passing a validation test means the idea survived a particular set of checks on past data. It does not mean the rule will work next year, and it is not investment advice. Markets change, costs change, and rules that many people use can stop working.
Where this fits
The TellHawk BaaS engine runs single-ticker strategies in walk-forward folds, scores each fold against buy-and-hold and a random-entry baseline, and applies a four-lens validation gate. The Strategy Lab replays multi-stock strategies point in time, charges costs on every trade and grades the result with nine evidence tests. Both are research tools; neither predicts the future.
Sources checked
Retrieved on 2026-10-12. Walk-forward testing and the cost arithmetic are method descriptions with invented numbers, not claims sourced to a dataset.
- Bailey, Borwein, Lopez de Prado, Zhu: Pseudo-Mathematics and Financial Charlatanism, Notices of the AMS, May 2014 (in-sample versus out-of-sample, overfitting)
- Library of Congress for the SEC: Annotated bibliography on the behavioral characteristics of U.S. investors, August 2010 (survivorship bias definition)
- SEC Form 10-Q instructions and Form 10-K instructions (filing deadlines)