When Optimisation Backfires
Optimisers love noise. Why the "optimal" portfolio is often the most fragile one.
This capstone warning ties the optimisation module to the bias module: mathematical optimisers, fed real-world data, frequently produce fragile, terrible portfolios. The “optimal” weights an optimiser spits out are often the worst ones to actually hold.
The fatal flaw: *optimisers maximise based on estimated inputs (expected returns, volatilities, correlations) — and those estimates are full of noise, so the optimiser ends up maximising the noise.* A mean-varianceThe square of standard deviation — dispersion of returns. optimiser willArranging how your wealth passes on after death. eagerly dump huge weight into whatever asset happened to look best in the sample (often due to luck or measurement error), producing extreme, concentrated, fragile allocations that collapse out-of-sample. This is overfitting wearing a portfolio costume — the same disease as a curve-fit backtestTesting a trading strategy on historical data.. Worse, optimisers are notoriously unstable: a tiny change in the input estimates can swing the “optimal” weights wildly, a sign the solution is built on noise, not signal. The practical wisdom: the mathematically optimal portfolio is usually not the practically best one. Robust, humble approaches — equal weighting, simple risk-based rules, or heavily-constrained optimisation — routinely beat unconstrained optimisers out-of-sample, precisely because they don’t over-trust noisy estimates. Optimise less; diversifySpreading money across assets that don’t move together to cut risk. and constrain more.
- The flaw — optimisers maximise on noisy estimates, so they amplify estimation error (overfitting in portfolio form).
- The symptom — extreme, concentrated weights that are wildly unstable to tiny input changes.
- The irony — the “optimal” portfolio is often the most fragile; simple equal-weight or constrained rules beat it out-of-sample.
- The fix — constrain weights, use robust/simple rules, and distrust any optimiser’s exact “optimal” answer.
ExampleFeed an unconstrained optimiser slightly noisy return estimates and it might say “putThe right, not the obligation, to buy or sell at a set price. 70% in this one asset that looked great in-sampleThe data a model was built and fitted on.” — a fragile bet that craters live. Nudge the input estimates by 1% and the “optimal” weights lurch to something totally different. A boring equal-weight portfolio, ignoring the noisy forecasts entirely, often quietly outperforms it out-of-sample.
Key takeawayOptimisers maximise on noisy estimates, so they amplify noise — producing extreme, unstable, fragile “optimal” portfolios that fail live (overfitting in portfolio form). The mathematically optimal is rarely the practically best; simple equal-weight or constrained rules often beat unconstrained optimisers. Optimise less, diversifySpreading money across assets that don’t move together to cut risk. and constrain more.
FAQs
So should I avoid portfolio optimisation entirely?
Not entirely — but treat it with deep skepticism and heavy constraints. Use optimisation’s *concepts* (diversification, risk balancing) while leaning on robust, simple constructions (equal weight, risk parity, capped weights) that don’t over-trust noisy return forecasts. The evidence is clear that simple, constrained approaches frequently beat unconstrained “optimal” solutions out-of-sample.