Experimental design: a practical guide

An experiment manipulates something and observes the effect, with everything else held as constant as the design allows. Three features are required, and a study missing any of them is quasi-experimental or observational.

The distinction between an independent and a dependent variable is worth restating precisely, because the terms get used loosely. The independent variable is the one you set: it is independent of the participant, because you decided its value. The dependent variable is the one you measure, and its value depends on what you did. If you did not set the value of a variable — if you merely recorded whether participants happened to smoke, or which school they attended — it is not an independent variable in the experimental sense, however your software labels it, and the study is observational.

Within-subjects designs are considerably more efficient, because the largest source of noise in most behavioural research is variation between people, and using each participant as their own control removes it entirely. A within-subjects design can often detect the same effect with a third of the participants.

Factorial designs extend the same logic to more than one manipulated variable. A 2 × 2 design crosses two factors with two levels each, producing four conditions, and it delivers three results for roughly the cost of one: two main effects and an interaction. The interaction is usually the reason to run it, since it answers whether one factor's effect depends on the other — a question a pair of separate experiments cannot address at all. The cost is that participant requirements grow with the number of cells, and interactions need appreciably more power to detect than main effects, typically around four times as much for the same effect size.

Random allocation is what licenses the causal claim. Because participants are assigned by chance, the groups differ only by chance at the outset — on everything, including variables you never measured and never thought of.

That last point is the whole value of it. Statistical adjustment can only control for variables you measured; randomisation balances the ones you did not.

One point that surprises people: randomisation does not guarantee balanced groups in any single study, and checking baseline balance with significance tests is actively discouraged in trial reporting. If allocation was genuinely random, any imbalance is by definition due to chance, so a significance test asks a question whose answer is already known. Report the baseline characteristics in a table so readers can judge whether any imbalance is clinically or substantively important, and adjust for a variable in the analysis only if that was pre-specified.

Note that simple randomisation with small samples will not necessarily produce balanced groups — with 20 participants, an 8 to 12 split is entirely possible. Block randomisation prevents this, and stratifying on a variable known to predict the outcome strongly is worth doing when the sample is small.

What you compare against determines what your result means. A treatment that beats nothing has demonstrated something much weaker than one that beats a credible alternative.

Ethics constrain this choice more than any other part of the design. Withholding an effective treatment to create a no-treatment arm is rarely acceptable where one exists, which is why active controls dominate in clinical research. A waitlist design is the usual compromise in psychological and educational work: everyone eventually receives the intervention, and the comparison is made before the waiting group starts. Note that this limits follow-up, since after crossover there is no longer an untreated group to compare against.

In a within-subjects design, whatever participants do first affects what they do next. Three distinct effects are worth separating.

Counterbalancing

Practice and fatigue are handled by varying the order across participants, so that the effects cancel out across the sample rather than accumulating in one condition.

A practical note on how many orders you need. Complete counterbalancing requires the number of participants to be a multiple of the number of orders, so with three conditions and six orders you want a multiple of six — 24, 30, 36. Planning a sample of 25 leaves one order over-represented, which reintroduces exactly the imbalance the counterbalancing was meant to remove. Decide the design first and let it constrain the target sample size, rather than the reverse.

Blinding matters because expectation affects outcomes on both sides. Participants who know they received the intervention may report improvement they would not otherwise notice; researchers who know may unconsciously alter how they administer a task or rate an outcome.

Record who was blinded, at what stage, and how the blinding was maintained, since a claim of double-blinding with no procedural detail is routinely queried.

Attrition deserves particular attention because it can undo randomisation entirely. If participants who find the intervention difficult drop out of that arm while the control arm loses nobody, the groups you analyse are no longer the groups you randomised, and the comparison has quietly become observational. Analysing by intention to treat — keeping everyone in the arm they were assigned to, whatever they actually received — preserves the randomisation, at the cost of diluting the estimated effect. Report both that and the per-protocol analysis where the two diverge.

Sample size is a design decision, not something to determine afterwards. A power analysis needs three inputs: the smallest effect worth detecting, the significance level, and the power you want.

The first and last rows show the efficiency of within-subjects designs: the same effect at the same power needs 128 participants between-subjects and 34 within-subjects. That difference is frequently decisive for a student project.

The effect size to plug in should be the smallest one worth detecting, not the one you hope to find or the one a previous study reported. Published effect sizes are systematically inflated by publication bias, so powering a study on them routinely produces one that is underpowered for reality. Asking “how small a difference would still change what anyone does?” gives a defensible figure and is the question examiners expect you to have asked.

1. No control group

A pre-post comparison cannot separate the intervention from maturation, history or regression to the mean.

2. Allocating by convenience

Assigning by class, session or willingness is not randomisation, and the groups will differ systematically.

3. No counterbalancing in a within-subjects design

Order effects then load entirely onto one condition and are indistinguishable from the effect you are testing.

4. Deciding the sample size afterwards

Stopping when a result reaches significance inflates the false positive rate substantially.

5. Selecting participants on extreme scores

Regression to the mean will produce apparent improvement with no intervention at all.

6. Ignoring differential attrition

If dropout differs between arms, randomisation is broken. Report attrition per arm and analyse by intention to treat.