Reliability and validity: a complete guide

Reliability is consistency. Validity is accuracy. A bathroom scale that reads three kilograms heavy every time is perfectly reliable and entirely invalid; one that fluctuates by five kilograms between weighings is neither.

The relationship between them is asymmetric and worth stating precisely. Reliability is a necessary but not sufficient condition for validity. A measure swamped by random error cannot be measuring anything consistently, so it cannot be valid. But a measure can be beautifully consistent and still capture something other than what you intended.

There is a further reason to keep the two separate in your write-up. Reliability sets a ceiling on validity that can be quantified: a measure's correlation with any criterion cannot exceed the square root of its reliability. A scale with an alpha of .64 can correlate at most about .80 with anything, however well designed. This is why a weak reliability figure is not merely untidy — it caps what the instrument can possibly demonstrate, and no amount of validity argument will lift that ceiling.

Test-retest needs a defensible interval

Too short and participants remember their answers, inflating the correlation; too long and genuine change gets counted as unreliability. Two to four weeks is conventional for stable traits. The right interval depends on how quickly the construct itself is expected to move, and you should justify the choice rather than default to it.

Split-half reliability is largely of historical interest now but explains where alpha came from. The idea was to divide a scale in two, correlate the halves, and adjust upwards for the fact that each half is shorter than the whole. The difficulty is that the result depends on how you split it — odd against even items gives a different figure from first half against second. Cronbach's alpha resolves this by being, in effect, the average of every possible split-half coefficient, which is why it superseded the older method.

Kappa has a known weakness worth anticipating, since a reviewer may raise it. When one category dominates heavily, kappa can be low even though the coders agree on almost everything — the so-called kappa paradox. The correction for chance becomes so large that little room is left for the statistic to move. Where this occurs, report kappa alongside the raw agreement and the marginal distributions, and note the imbalance rather than presenting the low kappa as though the coders had performed badly.

Why kappa rather than percentage agreement

Percentage agreement ignores the agreement expected by chance. If two coders each assign 90% of extracts to the same dominant category, they will agree about 82% of the time by chance alone, so 85% agreement is close to worthless. Running the numbers makes this concrete: with expected agreement of .82, an observed 85% gives κ = (.85 − .82) / (1 − .82) = .17 — slight agreement, not the substantial figure the raw percentage suggests. Kappa corrects for this, which is why it is the reportable statistic.

The ICC has several forms, and software will ask you to choose between them. The distinctions concern whether raters were selected at random or are the only ones of interest, and whether you want agreement in absolute terms or merely consistency in ranking. Report which form you used, because a two-way mixed model assessing absolute agreement and a one-way random model assessing consistency can give noticeably different values on the same data.

Convergent and discriminant validity are usually assessed together, because either alone proves little. A new measure of anxiety should correlate substantially with an established anxiety scale and only modestly with a measure of, say, extraversion. High correlations with everything indicate a general response tendency rather than a specific construct.

Construct validity deserves particular attention because it is the type examiners probe hardest and the one most often asserted without evidence. It is not established by a single analysis; the modern view treats it as the overarching concern that the others contribute to. Factor analysis showing the expected structure is one piece of evidence. So is the instrument behaving as theory predicts — scores rising after an intervention designed to raise them, or differing between groups known to differ on the construct. Each finding that matches prediction strengthens the case; each that does not weakens it, and should be reported rather than omitted.

If you use an existing validated instrument

If you develop your own

This is a substantial undertaking and is usually a study in itself rather than a preliminary step. The minimum credible sequence is: define the construct; generate an item pool from the literature; have experts review it for content validity; pilot on a small sample and revise; administer to a larger sample; examine the factor structure; compute internal consistency; and test convergent and discriminant validity against established measures.

Translating an instrument counts as modification, and a translated scale requires revalidation in the new language. The accepted procedure is forward translation by two independent translators, reconciliation, back-translation by a third who has not seen the original, and comparison — followed by a pilot to check that items read naturally and mean what was intended. Reporting that a scale was “translated into Urdu” without describing this process invites a straightforward challenge in a viva.

A 10-item scale measuring study confidence is administered to 240 undergraduates, alongside an established self-efficacy measure and a measure of extraversion. Two weeks later, 60 participants complete it again.

Read as a set, this is a credible case. The scale is internally consistent and stable, behaves as one factor, relates strongly to a conceptually similar measure and weakly to an unrelated one, and predicts a real outcome. No single row would establish much; the pattern across them is the argument.

Note also what is missing from that table and would strengthen it further: evidence that the scale behaves as theory predicts under intervention, and evidence that it performs comparably in a second, different sample. Validation is cumulative rather than a single study, and an honest write-up says which pieces of the argument are still outstanding.

Reliability and validity are concepts from measurement, and applying them unchanged to qualitative work misrepresents what that work claims. The established alternative framework is trustworthiness.

Note that inter-rater reliability still has a place in qualitative work, but a contested one. In content analysis, where the aim is systematic classification, kappa is appropriate and expected. In reflexive thematic analysis, where interpretation by the researcher is the instrument rather than a source of error, reporting a kappa misrepresents the method — and Braun and Clarke are explicit about this.

Triangulation deserves a note, since it is the trustworthiness criterion most often invoked and least often specified. There are several distinct kinds: data triangulation across sources or time points, investigator triangulation across analysts, theoretical triangulation across frameworks, and methodological triangulation across methods. Saying “triangulation was used” conveys almost nothing. Name which kind, describe what it consisted of, and say what it revealed — including where sources disagreed, which is usually the more interesting result.

1. Treating reliability as evidence of validity

A consistent measure can consistently measure the wrong thing. They are separate claims requiring separate evidence.

2. Citing the original study's alpha instead of your own

Alpha depends on your sample. Compute and report it for your data.

3. Reporting percentage agreement rather than kappa

Percentage agreement ignores chance agreement and overstates reliability substantially.

4. Modifying a validated instrument silently

Changing wording, response options or item order voids the original validation. Say what you changed and why.

5. Reporting alpha above .95 as excellent

Very high alpha usually indicates redundant items asking the same question repeatedly, not a superior scale.

6. Applying quantitative criteria to qualitative work

Use credibility, transferability, dependability and confirmability, and evidence each.