P-values: what they mean and what they don't

A p-value answers one precise question: if the null hypothesis were true, how often would I see a result at least this extreme? Nothing more, and in particular nothing about whether the null hypothesis is actually true.

The logic runs backwards from how most people read it. You begin by assuming there is no effect, work out what your test statistic would typically do under that assumption, and then ask how unusual your actual result would be in that world. If it would be very unusual, you have grounds to doubt the starting assumption.

This is why the null hypothesis is never “accepted”. The reasoning can only ever cast doubt on it. A large p-value means your data are compatible with the null — and quite possibly compatible with a great many other values too.

There is a further subtlety worth grasping, because it explains why so many published significant results fail to replicate. The p-value tells you how often data this extreme arise when the null is true, but it says nothing about how often the null is true in the pool of hypotheses your field tests. If most tested hypotheses are wrong — which is likely in exploratory fields — then a substantial share of significant results are false positives even when every test was run correctly. A field testing many long shots at α = .05 with modest power can easily end up with more false positives among its significant findings than true ones. This is not a flaw in the arithmetic; it is a consequence of applying a fixed error rate to a population of mostly-null hypotheses, and it is why replication rather than a single p-value is what settles a question.

The correct form of words

The .05 threshold has no mathematical basis. Ronald Fisher suggested it in 1925 as a convenient rule of thumb, remarking that it was for the researcher to decide what level of evidence to find persuasive. It stuck through repetition rather than through justification.

The genome-wide figure shows the principle at work: when you test a million hypotheses, a threshold of .05 would hand you 50,000 false positives, so the threshold has to move. The right α depends on how many tests you run and what a false positive costs — it was never a universal constant.

Statistical significance answers “is it distinguishable from zero?”. Practical significance answers “is it big enough to matter?”. These come apart constantly, and only the second is usually the question anyone cared about.

Because the p-value depends on both effect size and sample size, a large enough study will find almost any difference significant. A trial of 50,000 people can detect a blood-pressure reduction of 0.4 mmHg at p < .001 — a difference no clinician would act on.

A non-significant result means your data did not provide enough evidence to reject the null. It does not establish that the null is true. Distinguishing the two requires looking at the confidence interval.

Both scenarios produce p > .05 and are frequently written up identically, as “no significant difference”. That phrasing collapses a precisely estimated null result and an uninformative study into the same sentence, and it is one of the most common criticisms reviewers make.

The p-value is only valid if the analysis was specified independently of the result. Once you start making decisions after seeing the data, the stated error rate no longer applies.

A useful discipline when reading any significant result, your own included, is to ask three questions in order. How many tests were run to produce it? How large is the effect, in the units the reader cares about? And how wide is the interval around that estimate? A result that survives all three — one planned test, an effect of practical size, an interval that excludes the values you would consider negligible — is worth building on. A result that clears only the p-value threshold is worth repeating before it is believed.

How the false positive rate inflates

Run twenty independent tests on data with no real effects and you should expect roughly one significant result. Reporting that one and omitting the other nineteen produces a finding that looks solid and means nothing.

None of this makes the p-value useless. It remains a compact summary of how well the data sit with a specific null, and it is the reason significance testing became universal. The failure is one of exclusivity rather than of the statistic itself — reported alone, it is asked to carry conclusions it was never built to support.

Practices that invalidate a p-value

Corrections when multiple tests are genuinely needed

Four studies test the same intervention against the same control on the same outcome, measured in points. Each reports a difference, a p-value and a 95% confidence interval. The task is to say what each one establishes.

Study A — significant, but barely informative

The p-value clears .05 and the interval excludes zero, so the result is conventionally significant. But the interval runs from 0.04 to 9.56: the data are consistent with an effect so small as to be meaningless and with one that would be substantial. The honest reading is that the intervention probably does something, and this study cannot say how much.

Study B — significant, and reliably trivial

With 2,400 participants the estimate is precise: the true effect is somewhere between 0.19 and 1.01 points. It is almost certainly real, and almost certainly too small to justify the intervention. A p-value of .004 sounds far more impressive than a difference of 0.6 points deserves, which is exactly why the effect size and interval must accompany it.

Study C — non-significant, and uninformative

Here p = .180, so nothing is rejected. But the interval spans −2.5 to 12.9. This study cannot rule out a moderate harm or a large benefit. Writing it up as “no effect was found” would be a serious misstatement; it found nothing because it was too small to find anything.

Study D — non-significant, and genuinely null

Also non-significant, but for a completely different reason. The interval is [−0.45, 0.65], tight around zero. This study can say something positive: whatever the effect is, it is small. That is a real finding, and it is the only one of the four that supports a conclusion of no meaningful effect.

A result reported as t(78) = 2.94, p = .004, d = 0.66, 95% CI [0.21, 1.11] tells the reader that the effect is unlikely to be chance, moderate in size, and estimated with reasonable precision. The p-value alone tells them only the first of those three.