Systematic reviews: a step-by-step guide

A systematic review is defined by its method, not by how thorough it feels. The question, the search, the inclusion criteria and the analysis are all specified before the work begins, and documented so that someone else could reproduce them.

The effort involved is routinely underestimated, and it is worth being realistic before committing. A search returning 1,500 records means 1,500 titles and abstracts to screen, twice over if you are following the standard method, then perhaps 80 full texts to obtain and read, then extraction and appraisal for those that survive. For a lone researcher on a dissertation timetable, that is a substantial share of the available time and it front-loads onto the months when supervisors expect to see progress. Scope the question so the yield is manageable, and run a scoping search before writing the protocol so you know what you are committing to rather than discovering it afterwards.

Review types worth distinguishing

A systematic review can only be systematic if the question is specific enough that inclusion decisions follow from it. The usual scaffold is PICO.

The protocol states what you will do before you do it, which is what prevents inclusion criteria drifting to suit the studies you happen to find.

Writing the protocol is also the point at which most scoping problems surface. If you cannot write inclusion criteria precise enough for two people to apply consistently, the question is not yet specific enough, and no amount of effort later will compensate. Testing the draft criteria against a handful of papers you already know should be included is a quick and effective check.

Register it prospectively. PROSPERO is the usual registry for health and social care reviews and is free; OSF serves other fields. Registration is increasingly a submission requirement, and it protects you: a documented prior plan is the strongest answer to any suggestion that criteria were adjusted after seeing results.

The search is the part most often done badly, and it is the part that determines whether the review can claim completeness.

Sensitivity and specificity trade off against each other in a search exactly as they do in a diagnostic test. A broad search finds nearly everything relevant but returns a great deal of irrelevant material to screen; a narrow one is quicker but risks missing eligible studies. For a systematic review the convention leans heavily towards sensitivity, because a missed study is a flaw in the review while an extra thousand records is only labour. Where you have deliberately narrowed the search — excluding non-English publications, for example — state it as a limitation rather than leaving the reader to infer it.

The structure of a search

Each PICO element becomes a block. Within a block, every way of expressing the concept is joined with OR; the blocks are then joined with AND. For the example above, the intervention block might be (“retrieval practice” OR “testing effect” OR “practice testing” OR quiz*).

Publish the full strategy for at least one database as an appendix, exactly as run, including line numbers and the count returned at each line. Describing it in prose is not sufficient for reproducibility.

Screening is where most of the calendar time goes, and it is worth planning around that. A rate of roughly 200 to 400 titles and abstracts an hour is realistic once you are practised and the criteria are settled; full-text screening runs at perhaps eight to fifteen papers an hour depending on how easily eligibility can be judged. Multiply those against your expected yield before you commit to a timetable, because a search returning 4,000 records is several full working days of screening alone, twice over if it is done in duplicate.

Two reviewers should screen independently, with disagreements resolved by discussion or a third reviewer. Where resources do not allow full double screening, a common compromise is for a second reviewer to independently screen a random 20% and to report the agreement, usually as Cohen's kappa.

Including a study is not the same as trusting it. Risk of bias assessment is what separates a systematic review from a bibliography, and it must use an established tool rather than a checklist of your own.

Note that risk of bias is judged per outcome, not per study, in the current Cochrane tools. A trial may be at low risk for its primary outcome, which was objectively measured, and at high risk for a secondary one that depended on self-report from unblinded participants. Assessing the study as a single unit obscures that distinction, and it is the outcome-level judgement that should feed your synthesis.

Assess independently in duplicate, as with screening. Present the results per study and per domain, usually as a traffic-light figure, and — crucially — use them. An assessment that is reported and then ignored in the synthesis has served no purpose.

Design the extraction form before you start and pilot it on three or four studies. Redesigning it midway means revisiting every study already done.

Build the form in a spreadsheet or a purpose-built tool rather than in a word processor, and give every study a stable identifier you use consistently across screening, extraction and appraisal. Reconciling three separate files that refer to the same study as “Ahmed 2021”, “Ahmed et al.” and “study 14” is a predictable and entirely avoidable waste of a week.

Extract in duplicate where possible; extraction errors are common and consequential, particularly for the numbers that feed a meta-analysis. Where a paper reports a median and range but you need a mean and SD, use a documented conversion method and record that you used it.

Meta-analysis is one option, not the default. The right synthesis depends on whether the studies are similar enough to combine and whether they report data that can be pooled.

Concluding that meta-analysis is inappropriate is a legitimate and often correct outcome. A pooled number derived from studies that measured different things in different populations is worse than no pooled number, because it carries false authority.

PRISMA 2020 is the reporting standard, comprising a 27-item checklist and the flow diagram. Most journals require both, and the checklist should be submitted with page numbers.

1. An inadequate search

One database, no controlled vocabulary, no grey literature. This cannot be repaired later and is the most common reason for rejection.

2. No protocol or registration

Without a prior plan, there is no defence against the suspicion that criteria followed the findings.

3. Single-reviewer screening with no agreement check

Duplicate screening is the standard safeguard against error and bias. If resources genuinely prevent it, have a second person screen a random sample, report the agreement statistic, and state the limitation explicitly rather than leaving it unmentioned.

4. Risk of bias assessed but not used

If the assessment does not influence the synthesis or a sensitivity analysis, it was decorative.

5. A flow diagram whose numbers do not add up

Check every stage reconciles before submission.

6. Meta-analysing incomparable studies

A pooled estimate across different populations and outcomes has false precision. Narrative synthesis is the better answer.

7. A question too broad to complete

Run a scoping search first. If it returns 40,000 records, narrow the question before writing the protocol.