Does switching to Bayesian A/B testing actually change “ship vs. no-ship” decisions? To find out, I simulated a realistic low-traffic experiment. The results revealed a surprising insight into how Bayesian models survive conservative business estimates.
The setup and the “haircut”
In low-traffic environments such as B2B SaaS or geo-tests, Frequentist tests struggle. I simulated an experiment with N=300 per arm and a true +2.0% lift. A standard Frequentist test (p < 0.05) achieved a meager ~20% statistical power.
Bayesian methods solve this by pooling k past, similar experiments to construct an informative prior, massively boosting power.
But in practice, teams rarely accept historical lifts at face value. Stakeholders often apply a conservative “haircut” to the prior mean, slashing a 2.0% historical lift to 1.0%. This skepticism is highly rational for two reasons:
- Diminishing returns: past UI optimizations captured the easiest users, so the next iteration will likely yield less.
- Selection bias: geo-rollouts target the best-fit markets first, so subsequent markets will naturally perform worse.
If we artificially discount our prior mean, does the Bayesian method lose its power advantage?
To find out, I ran 1,000 simulations varying k from 1 to 20. For each, I applied an increasingly aggressive haircut, from 0% to 100%, to the prior mean. Crucially, the prior standard deviation remained untouched, reflecting the true statistical precision of the k experiments.
The surprising result
Informative priors are surprisingly robust to conservative estimates.
1. The 50% compromise is a huge win. With k=10, you can discount your prior mean by 50% and still achieve 35% power, against the Frequentist baseline of ~20%. You perfectly accommodate the team’s conservatism while gaining a +15% absolute power advantage.
2. More history builds a thicker shield. The black breakeven line slopes right. At k=3, a 50% haircut loses to the Frequentist baseline. But at k=20, the historical precision is so strong you can survive a massive ~65% haircut.
3. The 100% haircut is a trap. Setting the prior mean to zero while keeping the tight historical variance tells the model “I am highly confident the true effect is exactly zero”. This false belief drops Bayesian power below the Frequentist baseline.
4. k=1 is high-risk, marginal-reward. A single historical draw is highly vulnerable to sampling noise. Applying just a 25% haircut almost completely eliminates your power advantage. If you only have one past experiment, run a Frequentist test.
The bottom line
The conversation around historical data doesn’t have to be “trust it fully or don’t use it.” You can happily concede a 50% conservative haircut, and the math shows you will still run a more efficient, higher-powered experiment.
I’m eager to hear from the Bayesian practitioners out there. Does this align with how your team handles prior-skepticism in production?
Senior data scientist and PhD economist specializing in experimentation and observational causal inference. Previously Uber, Airbnb, Stripe, the Federal Reserve, and the SEC. More about my background.
New write-ups when I finish them
I simulate a method, run it against known ground truth, and publish what happened. No schedule, and no filler between posts.
Unsubscribe anytime.