Testing for normality: Shapiro-Wilk and the alternatives

A normality test asks whether your sample is consistent with having been drawn from a normal population. The null hypothesis is that it is, which reverses the usual logic: here a significant result is the unwelcome one.

Shapiro-Wilk is generally the most powerful of these and is the sensible default. Kolmogorov-Smirnov without the Lilliefors correction is genuinely poor — it assumes the population mean and standard deviation are known rather than estimated — yet it still appears in software menus and in student work.

Report the statistic as well as the p-value in every case, since the two carry different information and only one of them is independent of sample size.

The Shapiro-Wilk p-value confounds two things: how non-normal the data are, and how much data you have. This makes it unreliable at both ends of the range, and in opposite directions.

The irony is exact and worth stating plainly. In small samples, where normality matters most because the central limit theorem cannot help you, the test is least able to detect a problem. In large samples, where normality matters least because the central limit theorem does most of the work, the test is most likely to flag one.

Simulation work has made this concrete. For the same population with mild skew, a normality test may reject about 15% of the time at n = 20 and close to 100% of the time at n = 1,000 — while the t-test's actual error rate stays acceptably near its nominal 5% throughout. The normality test and the test you care about are answering different questions, and only the second one bears on whether your conclusion is sound.

A quantile-quantile plot compares your observed values against the values a normal distribution would produce. If the data are normal, the points fall along a straight line.

This diagnostic information is the reason plots beat tests. Knowing the data are “not normal” tells you nothing about what to do; knowing they are right-skewed points directly at a log transform, and knowing they have heavy tails points at a robust or rank-based method instead.

Histograms are worth plotting alongside, though they are less diagnostic than they look. The apparent shape of a histogram depends heavily on the bin width chosen, and the same data can look approximately normal or clearly bimodal depending on that arbitrary decision. A Q-Q plot has no such tuning parameter, which is why it is the more reliable of the two for judging normality specifically. Use the histogram to spot gross features — bimodality, floor effects, a spike at zero — and the Q-Q plot to judge the shape of the tails.

A great deal of unnecessary normality testing happens because people assume every test requires it. Most do not, and several that appear to actually require something subtly different.

For t-tests, ANOVA and regression, the normality assumption applies to the residuals — the part of each observation the model does not explain — not to the outcome variable as a whole.

This distinction has a practical consequence that regularly leads people astray. If two groups genuinely have different means, the combined distribution of the outcome will look bimodal or at least non-normal, even when each group is perfectly normal internally. Running Shapiro-Wilk on the pooled outcome variable will return a significant result, and it will be telling you about the group difference you are trying to test, not about a violated assumption.

The same logic explains a related confusion about predictors. Nothing in linear regression requires the predictor variables to be normally distributed. A binary predictor, a heavily skewed one, or a deliberately balanced experimental factor are all perfectly acceptable. The assumption concerns only the residuals, and testing your predictors for normality is effort spent on a requirement that does not exist.

Three studies report Shapiro-Wilk alongside their sample size. The correct response differs in each case.

Study C is the case that causes most unnecessary work. A W statistic of .997 indicates the data are almost exactly normal; the significant p-value reflects only that with 1,240 observations the test can detect a departure far too small to affect anything. Switching to a non-parametric test here would sacrifice power for no benefit.

Study A is the one that most often goes wrong in the opposite direction. With 18 observations the researcher sees a comfortable p-value, records “normality was confirmed”, and proceeds. But the Q-Q plot shows a real curve, and at that sample size the central limit theorem offers little protection either. This is precisely the situation where the assumption matters most and where the test is least capable of detecting a violation — so the plot should override the p-value entirely.

In order of preference, and only once you have established that the departure is real and that your test genuinely requires the assumption.

The first row of that table deserves more weight than it usually gets. The central limit theorem says that the sampling distribution of the mean approaches normality as the sample grows, whatever the shape of the underlying data. That is what the t-test and ANOVA actually require — not that your observations are normal, but that the mean behaves normally across hypothetical repeated samples. For moderately skewed data this holds well by about n = 30 per group, and for severely skewed data by perhaps n = 50. Above those thresholds, a significant normality test is usually telling you something true and irrelevant.

There is a real cost to switching unnecessarily. A rank-based test discards information about the spacing between values, and the conclusion it supports is subtly weaker — a statement about which group tends to score higher rather than about how much higher. If the parametric test was valid, you have given up both power and precision of interpretation for nothing.

Choosing a transformation

One caution about transforming to satisfy a test rather than to reflect the data. If a variable is skewed because it genuinely is — income, reaction times, hospital length of stay — a log transform is often the more natural scale to think on, and the transformation is substantively justified as well as statistically convenient. If it is skewed because of a handful of data-entry errors or a floor effect, transforming hides a problem that should be fixed or reported instead. Ask why the skew is there before deciding what to do about it.

1. Testing the raw outcome instead of the residuals

With genuinely different group means, the pooled outcome will look non-normal even when every group is normal.

2. Treating a significant result in a large sample as a problem

Check W and the Q-Q plot. With n over a few hundred, significance is close to guaranteed.

3. Treating a non-significant result in a small sample as reassurance

With n under 30 the test has very little power. It has not established normality.

4. Testing normality for tests that do not require it

Chi-square, Mann-Whitney and Kruskal-Wallis have no normality assumption whatsoever. Running Shapiro-Wilk before them is wasted effort that occasionally leads people to abandon a perfectly appropriate test.

5. Using Kolmogorov-Smirnov without the Lilliefors correction

The uncorrected version assumes known population parameters and is badly conservative. Use Shapiro-Wilk.

6. Switching to non-parametric on the strength of the p-value alone

Decide from the plot, the sample size and the robustness of the intended test — not from one number.