I’ve coached dozens of data scientists through case study interviews at Meta, Netflix, Amazon, and Uber. The pattern is striking: technically brilliant candidates who can design clustered experiments and explain causal inference fail because they rush past product sense.
The common mistake
Interviewer: “We’re considering a 20% discount coupon in the App Store. How would you measure impact?”
Many candidates immediately answer: “Run an A/B test with conversion rate as the primary metric.”
They’ve answered in 30 seconds. The interviewer isn’t impressed, because their team spent months on this problem. When you solve it instantly, you’re demonstrating naivety, in that you don’t see the complexity, and arrogance, in that their complicated problem apparently has a trivial solution.
What strong candidates do differently
After coaching clients to offers at OpenAI, DeepMind, Google, Meta, Uber, Airbnb, Stripe, and TikTok, from mid-level through senior staff, I consistently hear that deep product sense differentiated them.
Strong candidates pause and work through the shape of the problem. Who’s affected and how? What makes this require a data scientist at all? Are there temporal dynamics? Attribution challenges? Winners and losers? What are the strategic trade-offs?
For the coupon example: some users would have paid full price anyway, which is cannibalization, while others are truly incremental. Conversion rate is low variance and easy to detect, while revenue metrics are high variance and harder to detect, but conversions alone don’t tell the full story about profitability.
Strategic context matters too. Is this an established company optimizing profitability, or a growth company willing to sacrifice margin for user acquisition? There’s also a time-based dynamic worth raising around pull-forward purchases.
After establishing this foundation, the metrics and measurement strategy discussions flow naturally. You can’t choose the right metrics without understanding who’s affected and what complications exist.
Why I keep coming back to this
Across 2021, 2023, and Spring 2026 I have converted 13 of 16 final rounds, most recently taking senior offers at Google, Uber, and Meta and staff offers at Figma and Attentive. In April and May 2026 alone I sat more than 80 interview rounds across sixteen companies.
Product sense is the single most common gap I see, and it is the one that separates a technically correct answer from an offer. After explaining this framework in more than a hundred coaching sessions, I turned it into written material covering the complete approach: product sense, metrics, and measurement strategy, from vanilla A/B testing through geo-clustering and observational causal inference.
The takeaway whether or not you ever buy anything from me: invest time in product sense. That’s where strong candidates separate themselves.
Senior data scientist and PhD economist specializing in experimentation and observational causal inference. Previously Uber, Airbnb, Stripe, the Federal Reserve, and the SEC. More about my background.
New write-ups when I finish them
I simulate a method, run it against known ground truth, and publish what happened. No schedule, and no filler between posts.
Unsubscribe anytime.