hierarchical testing

A trial that asks several questions of the same data risks answering one of them wrongly by chance alone. The European Medicines Agency's guidance on multiplicity puts numbers on it: five independent tests, each run at a significance level of 2.5 percent, carry roughly a 12 percent chance that at least one comes back positive when there is nothing there to find.

Hierarchical testing is one answer to that. The trial's hypotheses are ranked by clinical importance and the ranking is written into the protocol before any data are analysed, so nobody can pick the most flattering result afterwards. Each hypothesis is then tested in turn at the full significance level, with no splitting of the threshold between them. The rule that makes the method work is what happens at a failure: once a hypothesis in the sequence cannot be rejected, no confirmatory claim can rest on it or on anything ranked below it.

For a reader, the order is a promise made in advance. A trial that clears its first test and misses its second has genuinely answered the first question, and the results further down the list are still reported, but they are described rather than claimed.