How to test a signal honestly

Finance is full of confident claims backed by charts that don't survive a second look. Our founding rule is that a signal earns its place only if there's evidence it predicts returns — so the hard part isn't finding claims, it's testing them in a way that can actually prove us wrong. Most of the work in this lab is exactly that: taking a popular idea, pulling the data ourselves, and seeing what's left when the easy mistakes are removed.

What it is

Honest testing means measuring a signal against the thing it's really competing with, over the periods it claims to work, on data we trust, with the failures left in. It's the opposite of finding a chart that agrees with you and stopping there. We'd rather kill a good-sounding signal than carry one we can't defend.

The traps we design around

The wrong benchmark. This one bites hardest. Our insider-cluster study looked value-destructive over 12 months — until we realised ~85% of that was just small companies trailing the S&P, not the signal. Measured against the right peer group (small-caps), the effect was honest and modest. Always ask: compared to what?
Survivorship. If you only test the companies that still exist, you've quietly deleted the ones that went to zero. We check whether the dropouts are bankruptcies or buyouts before trusting a result.
Believing the white paper. Especially in finance, a published edge is often gone by the time it's published, or never held net of costs. We pull the data ourselves rather than take the abstract's word for it.
Forcing a straight line. A signal can be strongly predictive in calm markets and anti-predictive in a regime shift. We look at the relationship in buckets across conditions, not as a single correlation number that averages the two into a comfortable lie.
Net of the real world. UK dealing costs and stamp duty quietly eat small, frequent edges. A signal has to clear the costs we'd actually pay, not a frictionless ideal.

How we use it

Every study reports the net effect plainly — including when the answer is "this doesn't work" or "only at one month." A reject decision is as valuable as a keep decision, and we publish both. The verdict on a signal can change as more data accrues; when it does, we say so and show why.

The honest caveat

No single test is proof. Honest testing lowers the odds of fooling ourselves; it doesn't eliminate them. That's why conviction comes from independent strands agreeing and a margin of safety underneath — so that being wrong about any one signal isn't fatal.

A plain-English explainer of how we think — part of our evidence-driven framework. Not investment advice.