Parameter Sensitivity
A good strategy works across a range of settings. If only one magic number works, run.
Parameter sensitivity analysis tests how your strategy performs as you vary its parameters (look-back lengths, thresholds, etc.) around their chosen values. It’s a powerful, simple check for whether you’ve found a real edgeA repeatable, structural reason your trades win over time. or fooled yourself.
The decisive test: *a robust strategy works across a range of parameter settings; an overfit one works only at a precise “magic number.” If your strategy returns 22% with a 50-day look-back, you should hope it also returns roughly 18–24% at 45, 48, 52 and 55 days — a smooth plateau* of good results around your setting means the edgeA repeatable, structural reason your trades win over time. is real and not knife-edgeA repeatable, structural reason your trades win over time. dependent. But if it returns 30% at exactly 50 days and collapses to near zero at 48 or 52, that’s a glaring red flag: you’ve found a spike — a lucky coincidence in the data, not a durable pattern. Real edges are broad and stable; overfit ones are narrow and fragile. So when choosing a parameter, don’t pick the single highest-returning value (that’s curve-fittingTailoring a strategy so closely to the past it fails on the future. the peak) — pick one in the middle of a wide, stable plateau, accepting slightly lower headline returns for far greater robustness. If only one magic number works, run.
- The test — vary each parameter around its value; check whether performance is a stable plateau or a fragile spike.
- Plateau = robust — good results across a range mean the edgeA repeatable, structural reason your trades win over time. isn’t knife-edgeA repeatable, structural reason your trades win over time. dependent (a real pattern).
- Spike = overfit — great only at one exact value, collapsing nearby, signals a lucky data coincidence.
- The rule — pick parameters in the middle of a wide plateau, not the single peak; “magic numbers” are a red flag.
ExampleStrategy A: ~20% return whether the moving averageA line that smooths price into its underlying trend. is 40, 45, 50, 55 or 60 days — a broad plateau; trustworthy, pick ~50. Strategy B: 35% at exactly 50 days but −2% at 48 and 52 — a lone spike; that 35% is almost certainly luck and willArranging how your wealth passes on after death. vanish live. A’s “lower” but stable return is the real edgeA repeatable, structural reason your trades win over time..
Key takeawayVary your parameters and check the shape: a robust edgeA repeatable, structural reason your trades win over time. shows a stable plateau of good results across a range; an overfit one shows a fragile spike at one magic number that collapses nearby. Choose parameters in the middle of a wide plateau, not at the peak — if only one exact value works, the edgeA repeatable, structural reason your trades win over time. isn’t real.
FAQs
Should I always pick the best-performing parameter value?
No — that’s curve-fitting the peak. Pick a value in the *centre of a broad region* of good performance, even if its headline return is a bit lower than the single best point. Robustness to parameter choice is strong evidence the edge is real; dependence on one precise number is strong evidence it’s overfit and will fail live.