Meta-analysis: a complete guide

A meta-analysis pools the results of several studies into one estimate, weighting each by how precisely it measured the effect. It answers a question no single study can: what does the evidence as a whole say?

The key word is weighting. A meta-analysis is not the average of the studies' findings. Each study contributes in proportion to 1 / variance — its inverse variance — so a large, tightly estimated trial counts for far more than a small noisy one.

It is worth being clear about what the pooled number represents, because it is not the answer to “does this work?” in any simple sense. Under a fixed effect model it estimates the single common effect assumed to underlie every study. Under random effects it estimates the mean of a distribution of true effects that vary across settings. Those are different quantities, and the second one carries an implication people often miss: if true effects genuinely vary, then the average may not describe any particular setting well, including yours. This is why the prediction interval matters more than the confidence interval for anyone deciding whether to adopt an intervention.

There is one further reason pooling is worth the effort. Individual studies are frequently powered to detect only large effects, so a genuine but modest effect produces a run of non-significant results that, read individually, look like consistent evidence of nothing. Pooled, those same studies can show the effect clearly. This is the pattern behind a number of interventions that were dismissed for years on the strength of underpowered trials and later established by meta-analysis.

Meta-analysis and systematic review are not the same thing

A systematic review is the process of finding, screening and appraising the studies. A meta-analysis is an optional statistical step at the end of it. Every meta-analysis should sit inside a systematic review; not every systematic review should end in a meta-analysis, and saying so is a legitimate and often correct conclusion.

The most consequential decision in a meta-analysis is made before any arithmetic: whether the studies are similar enough that a single pooled number means something.

Language and date restrictions deserve the same scrutiny. Excluding non-English publications is common and rarely justified beyond convenience; it introduces a bias whose direction you cannot know in advance. If you restrict, say so plainly as a limitation and, where feasible, report how many records were excluded on that ground alone.

Where several studies report the same outcome from overlapping samples — a common problem with large cohorts producing multiple papers — including both double-counts those participants and artificially narrows your interval. Choose one, usually the most complete, and record the decision.

One decision precedes all of this and is easy to overlook: what the unit of analysis is. If a study reports three outcome measures, or compares three intervention arms against one control, including all of them treats correlated results as independent and understates the true uncertainty. The standard responses are to select one outcome per study in advance, to combine the arms, or to use a multilevel model that accounts for the nesting. Whichever you choose, state the rule in the protocol rather than deciding once you see the data.

Studies rarely report results in the same units, so each must be converted to a common metric before pooling.

Note the third column. Ratio measures and correlations are not pooled directly, because their sampling distributions are skewed. They are transformed to a log or Fisher's z scale where the distribution is approximately symmetric, pooled there, and converted back for reporting. Software does this automatically, but it explains why the confidence intervals you get back are asymmetric.

Four studies report a standardised mean difference with its standard error. We will pool them under a fixed effect model.

Step 1 — weights

Each weight is the reciprocal of the variance, and the variance is the standard error squared. For study A: v = 0.10² = 0.0100, so w = 1 / 0.0100 = 100.

Step 2 — the pooled estimate

ḡ = Σ(w × g) / Σw = 69.59 / 182.88 = 0.3805

Step 3 — its standard error and interval

SE = √(1 / Σw) = √(1 / 182.88) = √0.005468 = 0.0739

95% CI = 0.3805 ± 1.96 × 0.0739 = [0.2356, 0.5255]

z = 0.3805 / 0.0739 = 5.15, giving p < .001.

Under random effects, the same four studies above yield a slightly larger pooled estimate with a noticeably wider interval, because the between-study variance is added to each study's own variance before weighting. That flattens the weights, giving the smaller studies more say.

Heterogeneity is the extent to which studies disagree beyond chance. It is not a nuisance to be minimised — it is often the most informative feature of the analysis.

I² is a proportion, not an absolute quantity, which is why it can mislead. A set of very precise studies with small real differences can produce a high I²; a set of imprecise studies with large real differences can produce a low one. Report τ² and the prediction interval alongside it.

One practical warning about Q. It is a significance test, and like any significance test its power depends on how much data it has — here, on the number of studies. With five or six studies, Q will frequently fail to reach significance even when real heterogeneity is substantial, and a non-significant Q is routinely misreported as evidence that the studies are homogeneous. It is not. With a large number of studies the opposite happens, and trivial differences reach significance. Never let Q alone determine whether you treat the studies as consistent.

Explaining heterogeneity

Where heterogeneity is substantial, the productive response is to investigate rather than to report it and move on. Subgroup analysis splits studies by a characteristic decided in advance; meta-regression relates the effect size to a study-level variable. Both need a reasonable number of studies — a common rule of thumb is at least 10 studies per covariate in meta-regression — and both are observational, so a subgroup difference is a hypothesis rather than a causal finding.

Studies with significant results are more likely to be published, published sooner and published in English. A meta-analysis of the published literature therefore risks pooling a biased sample of the evidence, and it will overestimate the effect.

Registered reports offer a further structural fix, though they address the primary literature rather than the review. Where journals commit to publishing a study on the strength of its protocol, before results exist, the incentive to suppress a null finding disappears. Reviews drawing on fields where registered reports are common face a less distorted evidence base to begin with, and it is worth noting in your discussion when that is or is not the case.

The strongest protection is procedural rather than statistical: register the review prospectively, search trial registries for unpublished work, include grey literature, and state in advance how bias will be assessed.

1. Pooling studies that are not comparable

Different populations, interventions or designs produce a number that describes nothing real. Justify combinability explicitly.

2. Vote counting

Counting how many studies were significant ignores sample size and effect magnitude entirely. It is not a meta-analysis.

3. Choosing the model after seeing the heterogeneity

Specify fixed or random effects in your protocol, before the analysis.

4. Reporting I² without τ² or a prediction interval

I² is a proportion and can mislead in both directions. Give the absolute measures too.

5. Double-counting participants

Multiple papers from one cohort inflate the weight of that sample and narrow the interval falsely.

6. Ignoring risk of bias in the included studies

Pooling poor studies produces a precise estimate of a biased effect. Appraise quality and run a sensitivity analysis excluding high-risk studies.