The Linear Trap: Why Double Machine Learning Beats Fixed Effects

I’ve been digging into Double Machine Learning (DML) recently to understand how it bridges the gap between machine learning flexibility and causal inference. I’m not a DML practitioner — I built this simulation to stress-test whether it actually changes decisions compared to methods I already use, like fixed effects OLS. The test case: estimating the causal impact of frame rate (fps) on watch time for a streaming service.

The results revealed something concrete about where traditional econometric methods break down when product experiences aren’t linear.

The setup

I simulated 25,000 viewing sessions where I controlled the true data generating process. The setup included two challenges data scientists face regularly.

The confounding. High-end devices natively deliver higher frame rates, and users inclined to watch more tend to choose high-end devices. Highly-rated shows also tend to prioritize streaming bandwidth. Both paths inflate a naive fps to watch time estimate.

The “knee” (non-linearity). Frame rate degradation doesn’t affect watch time uniformly. Above 30 fps there is a flat plateau with no degradation, between 24–30 fps a gentle decline, and below 24 fps a steep cliff. The 24 fps threshold is where the user experience deteriorates quickly.

I estimated the total watch time opportunity from a +10% fps improvement using four methods: Naive OLS, OLS with Fixed Effects (user and show dummies), DML with LOESS, and the Oracle (true DGP).

=================================================================
Opportunity Sizing: +10% fps across all sessions
=================================================================
Avg fps:  31.4  →  34.5
Sessions crossing 24 fps knee: 1,537

Method                              Coeff     Avg/session        Total
----------------------------------------------------------------------
Naive OLS (biased)                 0.792        2.49 min     62,140 min
FE OLS (user+show FEs)             0.262        0.82 min     20,544 min
DML LOESS (causal)                   n/a        0.51 min     12,762 min
True DGP                             n/a        0.56 min     13,989 min

The results

Naive OLS is a trap. It wildly overestimates impact because it conflates device and show quality with frame rate, a classic confounding story.

Fixed effects remove confounding but force a dangerous linear assumption. Adding user and show dummies drops the coefficient from 0.79 to 0.26 min/fps, so the confounding is gone. But a linear specification applies that rate uniformly to every session, including the ~86% of sessions already above 30 fps where the true marginal effect is zero. It overestimates value at high frame rates and concentrates too little uplift at the knee where it actually lives.

DML LOESS recovers the true shape. Cross-fitting Random Forest nuisance models and applying LOESS to the residuals lets the model capture the non-linear causal structure, assigning near-zero uplift to high-fps sessions and concentrating the benefit near the 24 fps knee. The result (12,762 min) lands within ~9% of the true oracle (13,989 min). The residual gap is a known LOESS tail artifact at high fps, not a fundamental flaw.

An aside on cross-fitting hygiene: a peer reviewer flagged that in panel settings, GroupKFold (keeping all sessions from a given user in the same fold) is better practice than standard KFold, since sessions from the same user share latent factors and their residuals are correlated. Worth knowing. In my simulation the difference was negligible, because user historical watch rate captured enough of the user-level signal that the split strategy didn’t matter. The more meaningful fix is to cluster your standard errors at the user level.

The bottom line

If you’re operating in an environment where treatments have non-linear thresholds, such as video quality, page load times, or pricing, fixed effects can be an oversimplified trap. Not because they fail on confounding, which they handle, but because a linear specification misattributes effects across the distribution. DML requires more computational work and rigorous cross-fitting, but it lets you recover the actual shape of the user experience.

I’m eager to hear from causal inference practitioners: does this align with how your team handles non-linear continuous treatments in production?