Here’s a problem I’ve never had a satisfying answer to: the metric you actually care about takes a year to observe, and nobody wants to hold a decision for a year. The usual move is to pick a short-term proxy, engagement or early retention, and hope it points in the same direction as the long-term outcome. Athey, Chetty, Imbens and Kang (2025, Review of Economic Studies) formalize something better, called the surrogate index. I’m not an expert in this method, so I built a simulation where I control the true data generating process and could watch it succeed or fail with the answer key in hand. In a clean setting it works startlingly well, and the interesting part is how early.
The setup: a subscription business where I know the answer
I simulated a Netflix-style subscription service. Each user has a hidden “fandom” level that drives both how much they watch and how likely they are to cancel each month: higher fandom, lower churn. On top of that, each user-month gets a content-fit shock that moves watch time and churn together. Churn is permanent, so once you cancel you’re gone.
The long-term outcome is months subscribed over a 12-month horizon, a customer-lifetime-value proxy. The experiment nudges fandom upward for treated users, which lowers their monthly churn hazard and accumulates into more subscribed months. Because I built the machine, I know the true 12-month effect: about half an extra month of subscription, 0.51 on a control base of 7.6 months.
The catch is that this effect takes time to become visible. In month 1, everyone in both arms is still subscribed, so a naive month-1 readout is exactly zero by construction. The retention curves only separate as churn compounds. Meanwhile, what I can see early are the surrogates: watch time, shows completed, and months active so far.
The method: borrow the future from historical data
The surrogate index needs two samples. First, historical data from users before the experiment, where the full 12-month outcome has already played out. On that sample I fit a model mapping early surrogates to the long-term outcome. Second, the live experiment, where only a few months have elapsed. I apply the historical model to each experimental user’s surrogates, imputing the 12-month outcome they haven’t finished generating, and take treated minus control on the imputed values.
In the linear case there’s a tidy interpretation: the estimate is each surrogate’s treatment effect in the experiment, weighted by how well that surrogate predicts the long-run outcome historically. Treated users watched more and completed more shows; the historical data says how many future subscribed months each unit of that behavior is worth; multiply and sum. The experiment’s own 12-month outcome is never touched.
mean_watch(T) average watch time over their active months ≤ T
mean_shows(T) average completed shows over their active months ≤ T
active_months(T) how many of the first T months they stayed subscribed
Y = β0 + βw·mean_watch(T) + βs·mean_shows(T) + βa·active_months(T) + ε
τ̂(T) = βw·Δwatch + βs·Δshows + βa·ΔactiveAt T=3, the fitted weights say an extra hour of average monthly watch time predicts +0.08 months of eventual subscription, an extra completed show +0.07, and an extra active month +4.14. Surviving the first three months is by far the strongest signal. The three contributions work out to 0.13 + 0.18 + 0.11 = 0.42. The naive readout, in this notation, is just Δactive with no weighting at all.
Two things improve mechanically as T grows. First, active_months(T) equals min(Y, T), because churn is permanent: a user who cancels in month 5 has their entire outcome sitting inside that feature from T=5 onward. Second, for survivors, a T-month average of watch time is a sharper proxy for latent fandom than a one-month snapshot. By T=12 the first mechanism has swallowed everything and the surrogate index equals the 12-month readout identically, so convergence at the horizon is forced. The real question is how much of the answer is already sitting in the features months before that.
One simplification worth flagging: the OLS treats active_months(T) as just another regressor and ignores structure I actually know. A user who churned at month 4 has a fully determined outcome of 4, and a survivor at month 6 can’t finish below 6, but the linear fit respects neither. In practice you’d use a two-part or discrete-time survival model. I kept it linear because the weighting stays readable.
The race to the 12-month answer
The naive raw readout just measures the outcome accrued so far. It starts half a month too low, because it is missing all the future churn divergence, and closes the gap only as the calendar runs out. Annualizing it helps some, but at month 6 it’s still off by roughly a third of the true effect. This isn’t sampling noise that more users would fix: early months simply carry a small share of the effect.
The surrogate index starts biased too. With one month of data it sits about 0.11 months low, because a single noisy month of watch time is a poor stand-in for latent fandom. But by month 4 the bias is gone for practical purposes, at −0.002, and 95% of simulated experiments land within about 0.08 months of their own eventual 12-month readout. You get the 12-month answer 8 months early.
Watching the agreement tighten
Each point is one simulated experiment, early surrogate estimate against its own eventual 12-month answer. At T=3 the correlation is 0.61, good enough to get the direction right and too loose to trust for any single experiment. By T=6 it’s 0.87 and the miss is typically under 0.05 months. By T=9 it’s 0.97.
The bottom line
If you’re in a setting like this one, with a stable user-level trait driving both short-term behavior and long-term retention, plenty of historical data where the long-term outcome has fully played out, and a surrogate-to-outcome relationship that’s roughly stable across the two samples, the surrogate index can compress a 12-month experiment into about 4 months with essentially no bias. The waiting tax is mostly optional.
But I want to be plain about what this simulation did not test. Fandom never changes, the monthly shocks are independent, and the surrogate-outcome relationship is close to linear. The assumptions are about as friendly here as they can be. A treatment that inflates watch time without changing underlying loyalty, a more aggressive autoplay say, would fool this estimator completely, and there’d be no warning in the data that it had. Treat these results as what the method can do under favorable conditions; your setting probably isn’t this friendly, and mine isn’t either.
I’m curious how practitioners handle this in production. If you’re using early proxies to call long-horizon experiments, do you ever validate them against completed experiments where you did wait, and how often do they hold up?
Senior data scientist and PhD economist specializing in experimentation and observational causal inference. Previously Uber, Airbnb, Stripe, the Federal Reserve, and the SEC. More about my background.
New write-ups when I finish them
I simulate a method, run it against known ground truth, and publish what happened. No schedule, and no filler between posts.
Unsubscribe anytime.