I’ve been building a simulation of uplift modeling to understand something that sounds obvious but isn’t always easily implemented: a positive average treatment effect from an A/B test does not mean you should roll the change out to everyone. I built this to stress-test the standard playbook (run the test, check if the average lift is positive, ship it to 100%) against a more targeted alternative, in a simulation where I control the ground truth and can check my work.
The test case: the kind of ranking decision e-commerce sites, hotel-booking platforms, and airlines all face in some form. A treatment group sees pricier options pushed above the fold, instead of everyone seeing the same “best for anyone” default.
The setup: a nudge that helps some people and hurts others
Imagine a default ranking that’s “best for anyone”: not personalized to the individual searcher, just based on average purchase probability across everyone. This is a simplified example and a simple simulation, not a description of any real system.
In the simulation, each search shows 10 listings above the fold, a mix of two property types: normal ($100/night) and premium ($150/night). The default ranking shows about 20% premium listings in that top 10, roughly whatever mix converts best on average. The treatment re-ranks the page so that mix becomes 50% premium.
Whether someone books, and what they book, depends on three things: the mix of what’s shown above the fold, their own underlying propensity to book at all, and how price-sensitive they are. That third piece is what makes the treatment a mixed bag:
- Price-insensitive searchers see more premium options and are just as likely, or more likely, to book, often the pricier listing. Revenue per user goes up.
- Price-sensitive searchers get some of the affordable listings they’d actually book pushed out of the top 10. Some of them don’t book at all. Revenue per user goes down.
I simulated 100,000 searchers under a randomized 50/50 experiment, with realistic covariates: past bookings, historical price paid, device, market, trip type. One of them, historical price, correlates fairly strongly with price sensitivity; the rest barely do, and even the strongest one leaves real uncertainty behind. No single covariate fully reveals who’s price-sensitive, so the model has to work for the rest. Baseline booking rate in the control group is about 37%, which I set deliberately to sit near the real baseline conversion rate in the field experiment I cite below.
The naive decision: what the A/B test says
Averaged across the whole experiment, the treatment group generated $4.54 more revenue per user than the control group. That’s a real, positive lift. Under the standard playbook, this is the point where you ship it to everyone.
The targeted alternative
Instead, I trained a simple T-learner (two random forests, one fit on the treated arm and one on the control arm, both predicting revenue from the same covariates) using only the experiment data. The difference between the two forests’ predictions, for a given searcher, is that searcher’s estimated treatment effect: how much more, or less, revenue we’d expect from showing them the premium-forward ranking.
I then applied this model to a separate, held-out population of 200,000 searchers and used it to decide who should see the premium-forward ranking: anyone with a positive predicted treatment effect, about 69% of this population. The difference from a real deployment is that I also know each person’s true effect here, so I can check this decision rule against ground truth. No real company gets to do that; I’m using it to verify the method works before trusting it.
Full rollout generates $53.36 per user on average. Treating only the ~69% of users the model predicts will benefit generates $54.55 per user, a $1.19 improvement, about 2.2% more revenue than shipping to everyone. That gap exists because the other 31% has a negative predicted effect: showing them a pricier top-of-page pushes them toward not booking at all, and blanket rollout forces the treatment on them anyway.
Checking the model honestly
A single dollar figure is easy to overstate, so I checked how good the underlying predictions actually were, person by person, against the true effect I built into the simulation.
The correlation between predicted and true individual effects is 0.73. That’s real signal, enough to make the targeting decision above pay off, but it’s visibly noisy: any single person’s predicted effect could be off by a wide margin.
Part of why: nobody, including this model, ever observes both outcomes for the same searcher. A model like this predicts by borrowing information from other searchers who share similar covariates, not by learning any individual’s actual response. This applies to every uplift model, not just this simple one, and it’s why the model is far more trustworthy in aggregate than it is for any single person’s exact predicted dollar figure.
This isn’t just a simulated toy problem, either. Athey, Keleher, and Spiess ran something structurally similar in a real field experiment: over 53,000 CUNY students were randomized into behaviorally-nudged reminders to renew their financial aid. Nudging everyone improved early filing by 6.4 percentage points on average over a 37% baseline, the same kind of clear, positive ATE that would trigger a full rollout under the naive playbook. Targeting half the students with their preferred policy still captured about 75% of that benefit.
That 75% figure needs a caveat, though. The paper is explicit that it found no evidence of negative treatment effects, and that sending reminders is cheap. If a nudge is nearly free and never hurts anyone, capturing 75% of the benefit by treating half the students is a worse outcome than treating everyone and capturing 100%. The actual case for targeting in a setting like theirs is a resource constraint, not harm avoidance. Cost-constrained targeting and harm-avoidance targeting are two separate problems that happen to use the same tool.
Bottom line
If you’re in a setting where a treatment plausibly helps some people and hurts others, a positive average effect from your A/B test only tells you whether to consider rolling something out. It doesn’t tell you who should get it. This simulation is a friendly environment: the treatment effect has real structure tied to observable behavior, and I get to check my answer against ground truth I built myself. Real data will be noisier and the model’s edge over blanket rollout will likely be smaller. But the direction of the result held up here.
Have you seen a case where the average effect was positive but rolling out to everyone was still the wrong call?
Senior data scientist and PhD economist specializing in experimentation and observational causal inference. Previously Uber, Airbnb, Stripe, the Federal Reserve, and the SEC. More about my background.
New write-ups when I finish them
I simulate a method, run it against known ground truth, and publish what happened. No schedule, and no filler between posts.
Unsubscribe anytime.