p-value

A p-value answers one narrow question: if the thing being tested made no difference at all, how often would an experiment like this one throw up a result at least as striking as the one actually seen? The US National Institute of Standards and Technology puts it formally as the probability of the test statistic being at least as extreme as the one observed, given that the null hypothesis, the assumption of no real effect, is true.

The number runs from 0 to 1, and smaller means more surprising. A p-value of 0.05 says a result this large would turn up about one time in twenty by chance alone, and 0.0001 says about one time in ten thousand. Researchers normally fix that threshold before they look at the data, and a result below it is what gets called statistically significant.

Two things a p-value does not tell you. It is not the probability that the treatment works, and it is not a measure of how big the effect is: a tiny and useless difference measured in a very large trial can still carry a very small p-value. It says only how poorly chance explains what was seen, which is why it is read alongside the size of the effect and the confidence interval rather than on its own.