Minimum Detectable Effect

Every test can see some effects and not others. J-PAL, the Abdul Latif Jameel Poverty Action Lab, defines the minimum detectable effect as the smallest effect that, if true, has a chosen chance of producing an estimate that is statistically significant at the chosen level. Below it, the lab says, we may not be able to distinguish an effect from zero even if it is real^[1]. A test that cannot see the lift you hope for is a test whose miss means little.

Several things set it. J-PAL describes the usual settings as a significance level of 0.05 and a power of 0.8, and lists the other inputs: the sample size, the variance of the outcome and the share of the sample given the change. A bigger sample, a smaller variance and an even split between the groups all lower the minimum detectable effect. Observations that are correlated within a cluster raise it, which in general increases the required sample size^[1].

There is no number without assumptions. NIST says the question of how many measurements to include has no correct answer without additional information or assumptions, and that the answer is influenced by the risk of rejecting a true hypothesis, the risk of accepting a false one and the standard deviation. For a two-sided test at 0.05 with a 0.10 risk of missing the shift, its table gives 43 measurements to detect a shift of half a standard deviation and 11 to detect a shift of one^[2].

The relation is a square. Kohavi and colleagues give the sample size as (4rσ/Δ)², for 95% confidence and 90% power, where r is the number of variants, σ the standard deviation and Δ the smallest difference to be found^[3]. They note that asking to detect a 20% change in conversion instead of a 5% change, a factor of 4, cuts the users needed by a factor of 16^[3]. Halve the lift you want to see and you need about four times the pages.

The same arithmetic applies to the groups inside a survey. QueryRev’s study reads 6,294 sites, and 1,339 of them, 21.3%, name at least one AI crawler in robots.txt^[4]. A share worked out within those 1,339 rests on about a fifth of the sites read, so it needs a larger difference to stand out than a share across all 6,294.

Work forwards from what you have. Take the weekly clicks per page, their spread from past weeks, the number of pages in each group and the weeks you can run. With a significance level of 0.05 and a power of 0.8, J-PAL’s formula gives the smallest effect the test could find^[1]. It shrinks as the sample grows and as the variance falls.

Or work backwards from the lift that matters. If the business needs a lift of 10% to make a change worth the cost, the formula gives the number of pages, or weeks, needed to find a lift of that size^[1]. Remember the square: halving the target about quadruples the pages^[3].

Check the assumptions and write the answer as a range. The figure rests on the spread of past data, on the groups being independent and on the metric behaving as it did before. Pages of one site share a domain, links and releases, so they are not fully independent. Treat the result as approximate.

Illustrative: an invented site, to show the method.

A coffee-equipment shop can change titles on 40 kettle pages, on 200 grinder pages or on all 1,100 product pages. Using eight weeks of past clicks for each set, it works out the smallest lift each could show at a significance level of 0.05 and a power of 0.8. The answers are approximate, so it writes them as ranges: a lift of about 22% to 32% for the kettle pages, 10% to 14% for the grinder pages and 4% to 6% for all product pages.

The business needs a lift of 10% to make the work worth doing. Forty pages cannot show that: a test on them would miss a 10% lift most of the time, and a miss would say nothing about the change. The grinder pages are borderline. The full catalogue could show it with room to spare, but a test needs untouched pages to compare with, and the shop will not change everything at once. It plans the grinder test for 12 weeks instead of eight, which should pull the range down, and it will report a miss as “too small to see”.