Robustness Tests for Trading Strategies

You built a strategy that looked good in the backtest, and now you're staring at a list of names like Monte Carlo, System Parameter Permutations, Walk-Forward Matrix and "Retest on additional markets." Which ones do you need, and in what order? By the end of this piece you'll know what each of these checks stands for and how they build on each other in a documented example.

Why the list of cross-checks is off-putting at first

Most traders run into robustness tests for the first time when a strategy already shows good numbers in the backtest and the software offers to check it further. Then a whole row of labels shows up, without it being clear right away what each one actually does. That's no accident: these checks aren't a single step, they're a toolbox with different tools for different questions.

What robustness tests check, and what they're meant to guard against

Robustness tests, also called cross-checks, are described as a built-in part of StrategyQuant. Their purpose: automatically check how a strategy behaves under pressure, for example when individual trades are skipped or parameter values are changed slightly. The idea behind this is simple: a strategy whose result collapses at small changes may have been fitted too closely to the available data.

One thing matters here: passing these tests doesn't prove a strategy isn't overfit, and a robust result under these simulated conditions is no guarantee of future gains. The tests show behaviour under certain disturbances, nothing more.

The four built-in checks at a glance

The vendor documentation describes the built-in robustness tests as a group of four checks: Monte Carlo simulations, System Parameter Permutations, a check of the Optimization Profile and a walk-forward matrix. According to the documentation, these four can be switched on or off together as a group to work against overfitting. The exact naming and arrangement in your version may differ slightly from this overview, but the principle of a bundled group stays the same.

Switching on these four checks together doesn't replace reviewing the results yourself. The group helps weed out obviously fragile strategies early, but it doesn't take over the job of looking at the surviving candidates yourself.

How the Robustness Tests Tool works with Monte Carlo

Behind the Monte Carlo check sits a simple mechanism: the Robustness Tests Tool repeats the test of a strategy with randomly changed input parameters and randomly changed data. These many repetitions produce a Monte Carlo simulation, whose result is reported along with confidence levels.

A confidence level isn't a promise. It says something about the spread of the simulated runs, not about what actually happens live. And a single Monte Carlo run isn't enough on its own as proof of robustness anyway, that's exactly why the idea of repetition exists.

Older documentation, which applied to versions up to 3.6.1, described the same basic principle with a fixed, user-set number of test runs: the strategy was then checked on how it reacts to small changes in inputs, historical data or other components. So the basic idea, introducing small changes and watching the reaction, isn't new, even if current versions name it or build it differently.

A simple example: retest on additional markets

One of the more intuitive checks is the retest on additional markets. Here the same strategy gets tested again on other markets, meaning other symbols and or other timeframes than the ones it was originally built and tested on.

This is a good starting point because the principle is easy to follow: a strategy whose logic is fitted only to the exact price path of a single symbol will often do worse on a different symbol. But passing this check isn't proof that the strategy works on any market or timeframe. It only shows how it behaved on the specific additional markets it was actually tested on.

An example funnel: how cross-checks build on each other

The list of checks looks long when you view it as a whole. The documentation includes an example of how individual cross-checks can build on each other in a sequence, so that not every strategy runs through every check at full depth.

In this example, a newly generated strategy is first tested at the fastest available precision. If it fails the global filters at that stage, it's automatically discarded, without any of the more demanding checks ever running. Only the strategies that survive this first step get passed on to more precise checks, such as a retest at higher precision and a Monte Carlo check that manipulates individual trades.

The scope of this example matters: it's one possible setup, not a mandatory sequence that every project has to follow exactly. And even surviving this whole funnel doesn't show that the strategy will work the same way live.

The older five-step path as background

Older documentation, also covering versions up to 3.6.1, described a five-step path for checking a strategy's robustness: first review the out-of-sample period, then test on additional symbols and timeframes, then run Monte Carlo based robustness tests, and finally review the resulting equity curves and a confidence level table in a dedicated robustness analysis view.

This sequence is useful as background because it shows that the core idea, applying several independent checks one after another, has been part of the tool for a while. As an exact guide for a current version it doesn't hold up: tab names and individual steps can look different in newer versions.

What these tests can show, and what they can't

Each of the checks described here rests on the same basic idea: run a strategy under simulated changes to past parameters, past trades or past data, and watch what happens. That's a useful filter for spotting strategies whose good backtest result was tied to very specific conditions.

What these tests don't deliver is proof for the future. A passed retest on additional markets, a good confidence level from a Monte Carlo simulation, or surviving a whole test funnel says something about behaviour under those specific, past conditions. It says nothing about how the same strategy will behave under future market conditions that haven't been observed yet.

This article is for education only and is not investment advice. Past results, including backtests, do not guarantee future results. Trading financial instruments involves risk, and you can lose money.