ANOVA: a complete guide, with a worked example

ANOVA compares three or more group means by splitting the total variation in your data into two parts: variation between the groups, and variation within them. If the between-group part is large relative to the within-group part, the groups probably differ.

The F statistic is the ratio of those two, each expressed as a mean square: F = MSₖₑₜ₣ₑₑₙ / MS₣ₓₜₕₓₙ. An F near 1 means the groups differ no more than the noise inside them already varies. A large F means they differ by more than noise explains.

It is worth being precise about what “variance” means here, because the name of the test misleads people into thinking it compares spreads rather than means. ANOVA does compare variances — but the between-groups variance it computes is driven entirely by how far apart the group means are. If the means were identical, the between-groups variance would be zero regardless of how spread out the individual scores were. So the test compares variances as a means to an end, and the end is a comparison of means. This is also why the F test and the t-test agree exactly when there are only two groups: with k = 2, F = t², and the two produce the same p-value on the same data.

Why not just run several t-tests?

Because the false positive rate compounds. Comparing four groups pairwise takes six t-tests, and at α = .05 the chance of at least one false positive rises to about 26%. ANOVA tests all groups in a single procedure at a single α, which is what keeps the error rate where you set it.

The between-versus-within distinction matters as much here as it does for t-tests, and for the same reason: if the same people are measured repeatedly, their scores are correlated, and a between-groups ANOVA that ignores this discards the pairing and loses substantial power.

Interactions, which are usually the point

A two-way ANOVA gives three results: two main effects and one interaction. The interaction asks whether the effect of one factor depends on the level of the other — whether a teaching method works better for some students than others, for example. When an interaction is significant, interpret it first: the main effects can be misleading on their own, because an effect that is strongly positive in one subgroup and negative in another may average out to nothing.

Sample size balance deserves particular attention here, because it interacts with the variance assumption in a way that catches people out. When group sizes are equal, ANOVA tolerates quite substantial differences in variance without much distortion. When they are unequal, the direction of the imbalance decides which way the test errs: a large variance paired with a small group makes the test too liberal, inflating false positives, while a large variance in a large group makes it too conservative. Neither is obvious from the output, which is why balanced designs are worth some effort at the recruitment stage.

Homogeneity of variance is the one that matters

ANOVA is fairly robust to non-normality, particularly with reasonable sample sizes, because the central limit theorem does much of the work. It is far less robust to unequal variances when group sizes are also unequal — that combination can push the actual false positive rate well above the nominal .05.

Note also that the normality assumption applies to the residuals, not to the raw data. Testing whether your outcome variable is normally distributed overall is the wrong check — with genuinely different group means, the combined distribution will look non-normal even when every group is perfectly normal.

Fifteen students are assigned to three revision methods, five per group. Exam scores follow.

The grand mean is (68.0 + 58.0 + 76.0) / 3 = 67.33.

Step 1 — sum of squares between groups

SSₖ = Σ nₖ(x̄ₖ − x̄)²

= 5(68.0−67.33)² + 5(58.0−67.33)² + 5(76.0−67.33)²

= 5(0.449) + 5(87.05) + 5(75.17) = 2.24 + 435.25 + 375.85 = 813.33

Step 2 — sum of squares within groups

Each group's deviations from its own mean, squared and summed: group A gives 90.0, group B gives 68.0, group C gives 90.0.

SS₣ = 90.0 + 68.0 + 90.0 = 248.0

Step 3 — the ANOVA table

dfₖ = k − 1 = 2 and df₣ = N − k = 12. Dividing each SS by its df gives the mean squares, and their ratio is F.

Step 4 — the decision

The critical value for F(2, 12) at α = .05 is 3.89. Our F of 19.68 far exceeds it, giving p < .001. At least one group differs.

Adding a second factor gives you three tests for the price of one design: a main effect for each factor, and an interaction between them. The interaction is very often the reason the study was run.

What an interaction means

An interaction says the effect of one factor depends on the level of the other. A revision method that helps strong students and hinders weak ones has an interaction with prior attainment, even if, averaged across everyone, it appears to do nothing at all.

Simple effects

When the interaction is significant, follow it with simple effects analysis: test the effect of one factor separately at each level of the other. Rather than asking “does method matter?”, you ask “does method matter for high-attainers?” and “does method matter for low-attainers?” — two questions with potentially different answers, which is the whole point.

Reading the ANOVA table

Reading only the first row would produce the conclusion “method made no difference”. The third row shows why that would be wrong: the method does have an effect, in opposite directions for the two groups, and averaging them out is what produced the null main effect.

A significant F licenses further comparison but does not perform it. Post-hoc tests compare pairs while controlling the error rate across the whole family of comparisons.

For our example, Tukey HSD would compare A with B, A with C and B with C. Given group means of 68.0, 58.0 and 76.0 with a within-group mean square of 20.67, all three pairs separate clearly — retrieval practice beats spaced practice, which beats massed practice.

One structural point about post-hoc procedures: the number of comparisons grows quickly, and each is another chance for a false positive. Tukey HSD controls the error rate across the whole family of pairwise comparisons rather than each one separately, which is why its p-values are larger than uncorrected t-tests would give. That is the correction working as intended, not a loss of sensitivity to apologise for.

For our example: η² = 813.33 / 1061.33 = 0.766 — the revision method accounts for about 77% of the variance in scores. That is implausibly large for real educational data, and reflects the tidy numbers chosen for the arithmetic rather than anything you should expect to see.

1. Running multiple t-tests instead of an ANOVA

Six pairwise tests across four groups gives a 26% chance of a false positive. Use one ANOVA.

2. Stopping at a significant F

The F tells you a difference exists somewhere. Without post-hoc tests you cannot say where, and the analysis is unfinished.

3. Running post-hoc tests after a non-significant F

The omnibus F is what protects the family-wise error rate. Running pairwise comparisons after a non-significant F reintroduces exactly the inflation the ANOVA was there to prevent. Planned contrasts specified in advance are the exception, since they do not depend on the omnibus result.

4. Ignoring unequal variances

With unequal variances and unequal group sizes, the real error rate can far exceed .05. Use Welch's ANOVA.

5. Testing normality of the raw data

The assumption concerns the residuals. If group means genuinely differ, the pooled distribution will look non-normal regardless.

6. Interpreting main effects when the interaction is significant

In a two-way design, a significant interaction means the main effects may be misleading. Interpret the interaction first, usually with simple effects.

7. Using a between-groups ANOVA on repeated measures

It ignores the correlation between a participant's own scores, throws away the design's main advantage and usually hides a real effect.