This one is aimed at analysts who evaluate experiments with an online calculator, which is a perfectly reasonable thing to reach for when you have one treated group and one control. The same answer is available in a few lines of code, and it is worth seeing the two line up before you need anything more complicated than this.
Running it as a regression also gives you something another person can audit and rerun, rather than a number you typed into a form on a website. The larger payoff arrives later: variance reduction with CUPED, or clustered standard errors for geo and switchback designs, are the same regression with a few more terms in it.
Colab runs in the browser with nothing to install, and the code is seeded so your numbers will match mine.
Read the transcript
Evaluating experiments is one of the core parts of the job, so I want to show you two different ways of evaluating a simple one. You will see how similar they are, and hopefully come away understanding how t-tests and linear regression relate to each other.
I start by generating the data. There is a group randomly assigned into treated and control. If you are treated, the probability of converting is 65 percent, and if you are not treated it is 60 percent. When the data is generated the sample lands very close to that: the untreated group converts at 0.599 and the treated group at 0.658.
If you have never evaluated an experiment before, an online calculator is perfectly adequate for something this simple, with one treated group and one control group. I plug in the sample size and the number who converted in each group and run the test. The test statistic comes out at 6.13 and the p-value is essentially zero, so you would say the treatment increased conversion by about 5 percentage points and that the result is statistically significant.
You can also do this with linear regression, which is what I reach for. We regress convert on treated, and the intercept comes back as 0.599 with a coefficient on treated of 0.0592, so just about 0.06. The t-statistic is 6.14 and the p-value is again essentially zero. Notice that the t-statistic and p-value are identical to what the difference-in-means test gave us, because that is exactly what this regression is doing. Running a regression on a single binary variable is a t-test for the difference in means.
Look at what the coefficients actually are. The intercept of 0.599 is precisely the mean of the untreated group. The coefficient on treated is 0.0592, and adding the intercept to it gives 0.658, which is precisely the mean of the treated group. So the intercept is the control mean, the coefficient is the difference between control and treated, and the t-statistic of 6.14 is the test of whether that difference is zero. That is why the answer comes out the same as the simple difference in means.