Guides / hypothesis testing
Hypothesis testing: a step-by-step guide
Hypothesis testing is a procedure, and its validity depends entirely on carrying out the steps in order. This guide sets out each stage, explains how to choose the test your design actually supports, and covers the errors, effect sizes and reporting conventions that turn a p-value into a defensible conclusion.
Written and reviewed by Hafiz Ahmad Tariq, Senior BiostatisticianHypothesis testing is a procedure for deciding whether sample data provide enough evidence to reject a stated assumption about a population. You specify a null hypothesis of no effect and an alternative, set a significance level, collect data, compute a test statistic and its p-value, and reject the null if the result would be sufficiently unlikely under it. The procedure is valid only if the hypotheses and the test are fixed before the data are examined.
Definition
What hypothesis testing is
Hypothesis testing asks whether your data would be surprising if there were genuinely no effect. You begin by assuming there is none, work out how your test statistic would behave under that assumption, and then judge how unusual your actual result would be.
The logic is deliberately conservative. It never establishes that an effect exists; it establishes only that the data are hard to reconcile with the absence of one. That asymmetry is why the conclusions are worded unevenly — you may reject the null, but you may only ever fail to reject it.
It is worth understanding why the procedure is built this way rather than the more intuitive alternative of testing the hypothesis you actually believe. The null is the only hypothesis specific enough to calculate with: “there is no difference” fixes a single value, zero, from which the behaviour of the test statistic can be derived exactly. “There is a difference” is compatible with infinitely many values, each implying a different distribution, so there is nothing definite to compute against. The awkward double negative in “fail to reject the null” is the price of that mathematical convenience.
The p-value is only the fourth of five steps, and its meaning depends entirely on the first two having been completed beforehand. A correctly computed p-value from an analysis chosen after seeing the data is still an invalid p-value.
Overview
The five steps, in order
| Step | What happens | When |
|---|---|---|
| 1 | State H₀ and H₁ | Before data collection |
| 2 | Set α, choose the test, decide one- or two-tailed | Before data collection |
| 3 | Collect data to the planned sample size | — |
| 4 | Check assumptions, compute the statistic and p | After collection |
| 5 | Decide, report the effect size and interval, interpret | — |
The p-value is the probability of a result this extreme given one pre-specified test. Every decision made after seeing the data — a different outcome measure, an added covariate, a dropped case, a few more participants — adds an opportunity for a chance result while the reported p-value continues to assume there was only one. The number stops describing what it claims to describe.
Hypotheses
Step 1: stating the hypotheses
The null hypothesis states there is no effect and always contains an equality. The alternative states there is one. Between them they must cover every possibility.
| Research question | H₀ | H₁ |
|---|---|---|
| Do two groups differ? | μ₁ = μ₂ | μ₁ ≠ μ₂ |
| Is the new method better? | μ₁ ≤ μ₂ | μ₁ > μ₂ |
| Are two variables related? | ρ = 0 | ρ ≠ 0 |
| Do three groups differ? | μ₁ = μ₂ = μ₃ | At least one differs |
Hypotheses written about sample means — “H₀: x̄₁ = x̄₂” — are wrong, because you can simply look at your sample and see whether they are equal. There would be nothing to infer. Hypotheses are always claims about the population you sampled from.
Setting up
Step 2: choosing alpha and the test
Alpha is the false positive rate you are willing to accept. The convention is .05, meaning a 1-in-20 chance of rejecting a true null, but it is a convention rather than a law and should be chosen for the situation.
| α | False positives | Typical use |
|---|---|---|
| .10 | 1 in 10 | Exploratory or pilot work |
| .05 | 1 in 20 | The default in most fields |
| .01 | 1 in 100 | When a false positive is costly |
| .005 | 1 in 200 | Proposed for novel claims |
Lower alpha reduces false positives but makes real effects harder to detect, raising the false negative rate. The two move against each other at any fixed sample size, and the only way to improve both is to collect more data.
A one-tailed test halves the p-value, so choosing one after seeing the direction of your result is p-hacking. It is defensible only when a result in the opposite direction would be treated exactly as a null result would be — that is, you genuinely would not report it. In most research that is not true, so two-tailed is the right default.
Test selection
Choosing the right test
| Design | Continuous outcome | Ordinal outcome | Categorical outcome |
|---|---|---|---|
| Two independent groups | Independent t-test | Mann-Whitney U | Chi-square |
| Two paired measurements | Paired t-test | Wilcoxon signed-rank | McNemar |
| Three or more independent | One-way ANOVA | Kruskal-Wallis | Chi-square |
| Three or more paired | Repeated measures ANOVA | Friedman | Cochran’s Q |
| Association between two | Pearson’s r | Spearman’s rho | Chi-square |
| Predicting an outcome | Linear regression | Ordinal regression | Logistic regression |
Two features override everything in that table. If the same participants contribute more than one observation, you need a paired or repeated-measures test. If observations are clustered — pupils within schools, patients within clinics — you need a multilevel model, whatever the outcome type, because ordinary tests treat correlated observations as independent and produce p-values that are too small.
Answer four questions about your design and measurement level and be taken straight to the test that fits, with the assumptions listed.
Use the test chooserExecution
Steps 3 and 4: collect and compute
Collect to the sample size you planned, and stop there. Checking the p-value as data accumulate and stopping when it drops below .05 — optional stopping — substantially inflates the false positive rate, in some scenarios to above 20%.
Sample size should already be settled by this point, from a power analysis conducted at the design stage rather than from convenience or from what a previous study happened to use. The calculation needs three inputs: the smallest effect you would consider worth detecting, the significance level, and the power you want — conventionally 80%. Note that the first of those is a judgement about your field and your question, not a statistical quantity, and it is the one people find hardest. Asking “how small a difference would still change what anyone does?” is usually the most productive way in.
Check the assumptions before the test, not after
Running the test, finding it non-significant, then testing assumptions in the hope of justifying a different test is analysis-dependent decision making. State in your protocol which test you will use and which alternative you will switch to if a named assumption fails.
Reporting
Step 5: decide and report
| Result | Decision | How to phrase it |
|---|---|---|
p ≤ α | Reject H₀ | “There was a statistically significant difference” |
p > α | Fail to reject H₀ | “There was insufficient evidence of a difference” |
p < .001Interpretation in plain language is the step most often skipped, and it is the one examiners weight most heavily. A results section that stops at “the difference was significant, t(38) = 2.31, p = .027” has reported a computation, not a finding. Adding “students using the new method scored about 6.5 marks higher on average, roughly two thirds of a standard deviation” turns it into something a reader can evaluate against what they already know about the subject.
Failing to reject is not the same as establishing that no effect exists. If your interval is wide, the honest conclusion is that the study could not tell — which is different from, and considerably more useful than, “no difference was found”.
Worked example
A worked example, start to finish
A tutor believes a new revision method improves exam performance. Forty students are randomly assigned to the new method or to conventional revision, twenty in each group.
Step 1 — hypotheses
H₀: There is no difference in mean exam score between students using the new method and those using conventional revision (μ₁ = μ₂).
H₁: There is a difference in mean exam score between the two groups (μ₁ ≠ μ₂).
Two-tailed, because a method that made performance worse would be reported as a finding rather than ignored.
Step 2 — alpha and test
α = .05. The outcome is continuous, there are two independent groups, so an independent samples t-test is planned. If Levene's test indicates unequal variances, Welch's correction will be applied; this fallback is recorded in advance rather than decided later.
Step 3 — collect
Data are collected to the planned n = 40 and collection stops there. No interim analysis is run.
Step 4 — check and compute
New method: M = 68.4, SD = 9.1. Conventional: M = 61.9, SD = 8.7. Histograms show no severe skew, and Levene's test is non-significant, so the pooled t-test proceeds.
The pooled standard deviation is sₓ = √[((19 × 9.1²) + (19 × 8.7²)) / 38] = 8.90.
SE = 8.90 × √(1/20 + 1/20) = 8.90 × 0.3162 = 2.815
t = (68.4 − 61.9) / 2.815 = 6.5 / 2.815 = 2.309, with df = 38, giving p = .027.
Step 5 — decide and report
Since p = .027 ≤ .05, the null is rejected. The effect size is d = 6.5 / 8.90 = 0.730, a medium-to-large effect, and the 95% confidence interval for the difference is 6.5 ± (2.024 × 2.815) = [0.80, 12.20].
The p-value establishes that the difference is unlikely to be chance. The interval shows the data are consistent with anything from a 0.8-point benefit — negligible — to a 12.2-point one, which would be substantial. A conclusion of “the new method works” is fair; a conclusion about how well it works is not yet supported.
Errors
The two ways to be wrong
| Error | What happens | Rate | Controlled by |
|---|---|---|---|
| Type I | Rejecting a true null — a false positive | α | Your chosen significance level |
| Type II | Failing to reject a false null — a false negative | β | Sample size and effect size |
Power is 1 − β, the probability of detecting an effect that genuinely exists, and the convention is 80%. A study with 30% power will miss a real effect seven times in ten, which means most non-significant results from small studies are Type II errors rather than genuine nulls.
Note also that the two errors are not equally visible. A false positive enters the literature as a published finding and may be cited for years; a false negative usually results in a study that is never written up at all. That asymmetry in visibility is part of why published effects tend to shrink on replication.
Increasing the sample is the only lever that reduces both error types at once. Once the data are collected, the trade-off between them is fixed and no analysis choice will improve it. Calculating power after a null result — observed or post-hoc power — adds nothing, because it is simply a transformation of the p-value you already have.
Our power analysis calculator gives the sample needed to detect the effect you care about at your chosen power and significance level.
Use the calculatorEffect size
Why the p-value is not enough
A p-value confounds two things: how large the effect is and how much data you collected. With a large enough sample, almost any difference becomes significant; with a small one, almost nothing does.
| Study | p | Effect | Honest conclusion |
|---|---|---|---|
| n = 40 | .048 | d = 0.65 | Promising but imprecise — replicate |
| n = 5,000 | .001 | d = 0.05 | Real but too small to matter |
| n = 30 | .210 | d = 0.48 | Underpowered, not evidence of no effect |
| n = 800 | .640 | d = 0.02 | Genuinely close to nothing |
Rows two and three are the ones most often misreported — the second written up as an important finding, the third as evidence of no effect. The effect size and confidence interval make the correct reading obvious in both cases, which is why they belong beside every p-value you report.
Pitfalls
Seven mistakes that invalidate the test
1. Choosing the hypothesis after seeing the data
The p-value assumes a pre-specified test. Fitting the hypothesis to the finding makes the number meaningless.
2. Optional stopping
Checking as data arrive and stopping at significance can inflate the false positive rate above 20%.
3. Switching to one-tailed after seeing the direction
It halves the p-value on the strength of a result already observed.
4. Running several tests and reporting the best
Twenty tests at α = .05 give roughly a 64% chance of at least one false positive. Correct, or label the analysis exploratory.
5. Using a test the design does not support
Paired data need a paired test; clustered data need a multilevel model. Neither is optional.
6. Treating p > .05 as proof of no effect
Check the confidence interval. A wide one means the study was uninformative.
7. Reporting significance without an effect size
Significance says an effect is detectable, not that it is large enough to matter.
Answers
Frequently asked questions
What are the steps of hypothesis testing?
State the null and alternative hypotheses; set the significance level and choose the test; collect data to the planned sample size; check assumptions and compute the test statistic and p-value; then decide, report an effect size and interval, and interpret. The first two steps must be completed before any data are examined.
What is the null hypothesis in hypothesis testing?
The statement that there is no effect, difference or relationship in the population. It always contains an equality and is the assumption the test provisionally adopts. The test measures how surprising your data would be if it were true, and rejects it if they would be sufficiently unlikely.
How do I choose the right statistical test?
Three features decide it: whether the outcome is continuous, ordinal or categorical; how many groups are being compared; and whether observations are independent or paired. Clustering overrides everything — nested data such as pupils within schools require a multilevel model whatever the outcome type.
What does it mean if p is greater than 0.05?
That your data did not provide enough evidence to reject the null at that threshold. It does not establish that no effect exists. Check the confidence interval: a narrow one near zero suggests any effect is small, while a wide one means the study was simply unable to tell.
Should I use a one-tailed or two-tailed test?
Two-tailed in almost all research. One-tailed halves the p-value and is defensible only when a result in the opposite direction would be treated exactly as a null result would be. The decision must be made and justified before seeing the data.
What is the difference between Type I and Type II errors?
A Type I error rejects a true null — a false positive, at rate alpha. A Type II error fails to reject a false null — a false negative, at rate beta. Lowering alpha raises beta at any fixed sample size; only a larger sample improves both.
Why do I need an effect size as well as a p-value?
Because the p-value confounds effect magnitude with sample size. A trivial difference reaches significance in a large enough study, and a substantial one fails to in a small study. The effect size and confidence interval separate the two and are what make a result interpretable.
Can I test my assumptions and then choose a test?
Only if the rule was set in advance. Specify in your protocol which test you will use and which alternative you will switch to if a named assumption fails. Running the test, seeing the result and then looking for grounds to switch is analysis-dependent decision making and invalidates the p-value.
Send the data. Get a fixed quote.
Attach your dataset or just describe the project. A named statistician replies with a price and a deadline, usually within one working day.