Find me on

CUPED below the grain: predicting bookings when you can only randomize markets

In a simulated market-level test, a CUPED covariate built from booking-level predictions removed about half the variance, versus 28% from market history.

A common way to apply CUPED is to adjust each randomized unit by its own pre-period value of the metric, such as a user’s past spend or a market’s past conversion rate. A recent post from Instacart, “Variance Reduction Below the Randomization Grain” by Sergio Camelo, Caitlin Kearns, Matias Cersosimo and Tilman Drerup, describes a different covariate for tests randomized by region or market. Instead of using the cluster’s history, they predict the outcome for each order inside the cluster and average those predictions up to the level that was randomized.

To build personal intuition, I simulated a geo-based Airbnb experiment with a known ground-truth effect. Standard CUPED removed about 28% of the variance, while booking-level CUPED removed about half. Since the covariate is built from bookings during the experiment, I need to assume the treatment doesn’t change which bookings come in. The full write-up is on Substack, and the notebook below has all of the code.

Colab runs in the browser with nothing to install, and since the data is fixed and the model is seeded, your numbers should match mine up to small differences across library versions.

← All tutorials

The Data Science Interview Playbook

If you’re getting interviews but not offers, the Playbook covers how I prepare for each round, based on more than 80 interview rounds in Spring 2026.