Statistical Significance

The p-value is the probability of a test statistic at least as extreme as the one observed, given that the null hypothesis is true. NIST’s handbook says it is good practice to decide in advance of the test how small a p-value is required, which is the same as choosing a significance level, and that a level of 0.05 means the null hypothesis is rejected 5% of the time when it is in fact true^[1]. It adds that the choice of level is somewhat arbitrary^[1].

The American Statistical Association published six principles on p-values on 7 March 2016. Three matter to a search team. A p-value does not measure the probability that the studied hypothesis is true, or that the data were produced by chance alone. It does not measure the size of an effect or the importance of a result. And business decisions should not rest solely on whether a p-value passes a threshold^[2].

Controlled experiments use the same test. Kohavi and colleagues say the confidence level is commonly set to 95%, which implies that 5% of the time one will wrongly conclude there is a difference when there is none^[3]. They add that an A/A test, with the same experience for both groups, should reject the null hypothesis about 5% of the time^[3]. A threshold of 0.05 is a promise to be wrong now and then.

Many comparisons make that promise cheap to break. A report that checks hundreds of queries and page groups will find some that cross the line by chance. The ASA’s release names the habit: its president describes practices called p-hacking and data dredging that emphasize the search for small p-values over other statistical and scientific reasoning. The fourth of its principles says proper inference requires full reporting and transparency^[2].

A floor on group size is a related kind of restraint. QueryRev’s study charts a country-code ending once at least 50 of its sites were read, and gives the number of sites read beside each share; .it, at 44.6% blocking an AI crawler by name, and .cn, at 3.7%, are the highest and the lowest of the endings that meet the rule^[4]. A floor is not a test of significance, but it keeps a share from being read as more than the sites behind it can bear.

Set the test before the data are read: the metric, the comparison, the period and the threshold, often 0.05. NIST advises deciding in advance how small a p-value is required^[1]. Then compute the p-value from the difference between the groups and the noise in the metric.

Read it together with the size of the effect and the range around it. Because significance does not measure the size of an effect or the importance of a result^[2], a very small lift can be significant on a large site and matter little, while a large lift on a few pages can miss the threshold.

Read a miss as silence, not as a verdict. A result that does not reach the threshold may come from a test with too few pages or visits to see the effect. A range that includes zero says that no effect cannot be ruled out, and also that a gain cannot.

Illustrative: an invented site, to show the method.

A coffee-equipment shop rewrites the titles of 100 kettle pages and holds back 100 matched pages as the control. After eight weeks the difference in clicks between the two groups is a modeled range of 1 to 9 points above the control, with a p-value of 0.02: below the threshold of 0.05 the shop set in advance. The clicks result is significant.

The same pages’ revenue is a different story. Its difference from the control is a modeled range of −$0.7k to $2.3k a month, with a p-value of 0.31. The range crosses zero, and the threshold is missed. The shop reports both results in the same table: significant for clicks, not for revenue. It does not report the revenue as no effect, because a miss is not proof of none.