Delving deeper into some pitfalls
Theoretically, a single control can be shared across multiple treatments. The theory also says that a larger control can have benefits in terms of reducing the variance.
Assuming equal variances, the sample size of a two-sample t-test is given by 1 _ 1 _ N T + 1 _ N C, which translates into the harmonic mean of the sample sizes. When one has one control with x users, k equally sized treatments with size 1 − x _ k , the optimal control size is given by minimizing the sum k _ 1 − x + 1 _ x .
The solution is x = 1 _ √ _ k + 1. For example, if you have three treatments, the optimal control size is not 25% but 36.6%, and the optimal treatment size is 21.1% each. With k = 9, the control should get 25%, and each variant only 8.3%.
However, one needs to be careful, in practice, of the following...
