Guides  /  measurement

Inter-rater reliability

Cohen's kappa, weighted kappa, Fleiss' kappa and the intraclass correlation, worked through with real numbers. Including the result that catches most people out: two raters agreeing on 90% of cases can still produce a kappa of 0.44, and why that is the statistic behaving correctly rather than failing.

Tara Sutaria Written and reviewed by Tara Sutaria, Senior Statistical Analyst
Updated 4 November 202615 min read
What is inter-rater reliability?

The extent to which two or more independent raters assign the same value to the same thing. It is reported with a coefficient that corrects for the agreement you would expect by chance, because raw percentage agreement always flatters. For two raters and unordered categories, use Cohen's kappa. For ordered categories, weighted kappa. For three or more raters, Fleiss' kappa. For continuous ratings, the intraclass correlation coefficient. Percentage agreement on its own is not an acceptable answer in a methods chapter.

The problem

Why percentage agreement is not enough

Two reviewers screen abstracts for a systematic review. One includes 5% of what it sees, the other includes 5%, and they work entirely independently with no thought given to the content.

They will still agree about 90% of the time, because both are saying "exclude" to almost everything. Percentage agreement rewards them for a decision pattern that contains no shared judgement at all.

A chance-corrected coefficient asks a different question: how much of the observed agreement exceeds what the raters' own marginal habits would produce by themselves?

κ = (pₒ − pₑ) / (1 − pₑ)

Here pₒ is observed agreement and pₑ is expected agreement under independence. The denominator is the room for improvement that chance leaves available, so kappa reads as the proportion of that room the raters actually occupied.

✓κ = 1 is perfect agreement.
✓κ = 0 means agreement no better than chance.
✓κ < 0 is possible, and means systematic disagreement, usually a coding sheet applied in opposite directions.

Choosing

Which coefficient to use

RatersDataUseNote
2Nominal categoriesCohen's kappaThe default for two coders
2Ordered categoriesWeighted kappaNear-misses count partially
3+Nominal, same raters throughoutFleiss' kappaAlso works with different raters per subject
2+Continuous or intervalICCPick the form deliberately
2+Any, with missing dataKrippendorff's alphaHandles gaps and mixed levels
2Continuous, measuring agreementBland-AltmanShows bias and limits, not a single number
Pearson's r is not an agreement statistic

If rater B scores every case exactly three points above rater A, the correlation is 1.00 and the agreement is nil. Correlation measures whether the two move together, not whether they land in the same place. This is the single most common substitution in submitted work, and it always overstates reliability.

The benchmark table, and its limits

κLandis and Koch label
< 0.00Poor
0.00 – 0.20Slight
0.21 – 0.40Fair
0.41 – 0.60Moderate
0.61 – 0.80Substantial
0.81 – 1.00Almost perfect

These labels are conventions proposed in 1977, not statistical thresholds, and the original authors described them as arbitrary. What counts as acceptable depends on the stakes. A kappa of 0.65 is reassuring for exploratory coding of interview transcripts and inadequate for a diagnostic decision that determines treatment.

Worked example one

Cohen's kappa

Two reviewers independently screen 100 abstracts as include or exclude.

B: includeB: excludeTotal
A: include401050
A: exclude64450
Total4654100

Step 1. Observed agreement

Add the diagonal and divide by the total.

pₒ = (40 + 44) / 100 = 0.840

Step 2. Expected agreement

For each category, multiply the two raters' marginal proportions, then add across categories.

include: (50/100) × (46/100) = 0.50 × 0.46 = 0.2300
exclude: (50/100) × (54/100) = 0.50 × 0.54 = 0.2700
pₑ = 0.2300 + 0.2700 = 0.5000

Half the agreement here is attributable to the raters' own habits alone.

Step 3. Kappa

κ = (0.840 − 0.5000) / (1 − 0.5000) = 0.3400 / 0.5000 = 0.680

Observed agreement of 84% becomes a kappa of 0.680. Substantial on the conventional scale, and notably less impressive than the 84% would have sounded.

Step 4. The confidence interval, which is usually omitted

SE = √(pₒ(1 − pₒ) / (n(1 − pₑ)²))
= √(0.840 × 0.160 / (100 × 0.25)) = 0.0733
0.680 ± 1.96 × 0.0733 = 0.536 to 0.824

The interval spans "moderate" to "almost perfect". Reporting the point estimate alone implies far more precision than 100 abstracts support, which is why the interval belongs in the write-up and not just in the output.

Step 5. Look at the off-diagonal

Ten cases where A included and B excluded; six the other way. The disagreements are reasonably symmetric, so neither reviewer is systematically more inclusive. An asymmetric pattern, say 15 against 1, is a different finding: it points to one rater applying a looser threshold, and the fix is a calibration meeting rather than a larger sample.

The thing that catches people

The kappa paradox

Two raters screen 100 records for a rare condition.

B: yesB: noTotal
A: yes5510
A: no58590
Total1090100

pₒ = (5 + 85) / 100 = 0.900
pₑ = (0.10 × 0.10) + (0.90 × 0.90) = 0.0100 + 0.8100 = 0.8200
κ = (0.900 − 0.8200) / (1 − 0.8200) = 0.0800 / 0.1800 = 0.444

Ninety per cent agreement, and a kappa of 0.444. "Moderate", on a table where the previous example scored "substantial" on lower raw agreement.

This is not a flaw

When 90% of cases are in one category, two raters guessing would agree 82% of the time. There is only 18 percentage points of agreement left for genuine skill to claim, and the raters claimed 8 of them. Kappa is telling you the truth: on the small number of positive cases, which is the only part that matters clinically, the two raters agreed 5 times out of 15.

Quantifying it

Two indices separate the causes.

prevalence index = |(a − d)| / n = |5 − 85| / 100 = 0.80
bias index = |(b − c)| / n = |5 − 5| / 100 = 0.00

A prevalence index of 0.80 is extreme, and that alone explains the low kappa. A bias index of 0 confirms the two raters are equally inclusive, so there is no systematic skew.

PABAK, with a caveat

The prevalence-adjusted bias-adjusted kappa removes both effects.

PABAK = 2pₒ − 1 = (2 × 0.900) − 1 = 0.800

PABAK is a function of observed agreement and nothing else, so it reintroduces exactly the flattery that kappa exists to remove. Report it alongside kappa and both indices, never instead of kappa, and say what each one is doing. The honest write-up gives kappa 0.444 with a prevalence index of 0.80 and explains that agreement on positive cases was 5 of 15.

The design fix

If the category is rare, over-sample positive cases for the reliability exercise. Deliberately assembling a set of 50 likely-positive and 50 likely-negative records gives the coefficient something to work with. State that you did so, because the resulting kappa then describes agreement on a balanced set rather than on the natural case mix.

Worked example two

Weighted kappa for ordered categories

Two clinicians grade 100 scans as mild, moderate or severe. These categories are ordered, so mild against moderate is a smaller error than mild against severe, and unweighted kappa treats both as total disagreement.

B: mildB: moderateB: severeTotal
A: mild258235
A: moderate630945
A: severe171220
Total324523100

Step 1. Unweighted first

pₒ = (25 + 30 + 12) / 100 = 0.670
pₑ = (0.35)(0.32) + (0.45)(0.45) + (0.20)(0.23) = 0.1120 + 0.2025 + 0.0460 = 0.3605
κ = (0.670 − 0.3605) / (1 − 0.3605) = 0.484

This discards the information that most disagreements were one category apart.

Step 2. Apply weights

Assign a weight to every cell for how much credit a near-miss deserves. With k categories indexed from 0:

linear: w = 1 − |i − j| / (k − 1)
quadratic: w = 1 − ((i − j) / (k − 1))²

For three categories, a one-step miss earns 0.50 under linear weights and 0.75 under quadratic. A two-step miss earns 0 under both.

Step 3. Recompute with weighted sums

Replace the diagonal-only sums with weighted sums across every cell.

linear: Pₒ = 0.8200, Pₑ = 0.6080, κₓ = 0.541
quadratic: Pₒ = 0.8950, Pₑ = 0.7318, κₓ = 0.609

Step 4. Reading the three figures together

WeightingκInterpretation
Unweighted0.484Exact agreement only
Linear0.541Half credit for one-step misses
Quadratic0.609Three-quarter credit for one-step misses

The ordering is not a finding. More generous weights produce higher coefficients by construction, so a paper reporting 0.609 without naming the weighting scheme has reported a number that cannot be interpreted. Quadratic weights are conventional in medical imaging; linear weights are more conservative. Choose before you look at the result, and say which you chose.

Quadratic weighted kappa equals the ICC

For two raters and ordered categories treated as equally spaced, quadratic weighted kappa is numerically identical to the two-way intraclass correlation. That equivalence is useful: it means the choice between them is about which framing your readers expect, not about which answer is correct.

Beyond two raters

Three or more raters, and continuous ratings

Fleiss' kappa

Four raters classify 10 subjects into three categories. Fleiss' kappa handles any number of raters and does not require the same raters to assess every subject.

Agreement is computed per subject from how concentrated the ratings are, then averaged.

Pᵢ = (Σnᵢⱼ² − n) / (n(n − 1))

For a subject where all four raters agreed: (16 − 4) / (4 × 3) = 1.00. For a 2-2 split: (8 − 4) / 12 = 0.333. For a 3-1 split: (10 − 4) / 12 = 0.500.

Across the ten subjects:

category proportions: 0.375, 0.375, 0.250
P̄ = 0.6500
Pₑ = 0.375² + 0.375² + 0.250² = 0.3438
κ = (0.6500 − 0.3438) / (1 − 0.3438) = 0.467

Note what this does not tell you: which rater is the outlier. Fleiss' kappa is a single summary across the whole panel. To find the problem rater, compute pairwise Cohen's kappas and look for the row of low values.

The intraclass correlation

For continuous ratings. Three raters score 8 subjects on a 10-point scale.

MS_subjects = 6.661, MS_raters = 0.292, MS_error = 0.339

ICC(2,1) = (MSₛ − MSₑ) / (MSₛ + (k−1)MSₑ + k(MSᵣ − MSₑ)/n) = 0.863
ICC(2,k) = (MSₛ − MSₑ) / (MSₛ + (MSᵣ − MSₑ)/n) = 0.950

Two numbers from one dataset, and the gap between them is the whole point.

FormValueReport when
ICC(2,1), single rater0.863One rater will score each case in real use
ICC(2,k), average of k0.950The mean of all three raters is the score used

Reporting 0.950 when the instrument will be used by a single assessor overstates its reliability, because averaging three raters cancels error that one rater cannot. This is the most common ICC error and it is invisible in the output, since software prints both and labels them in small text.

The other choice is the model. ICC(1) treats raters as a random sample and no rater effect is estimated; ICC(2) treats raters as random and estimates their effect; ICC(3) treats these specific raters as the entire population of interest and so does not generalise to other raters. For a reliability study intended to support general use of an instrument, ICC(2) is almost always right. Related material sits in the reliability and validity guide and, for internal consistency rather than agreement, the Cronbach's alpha guide.

Doing it

In R, SPSS and Stata

R

library(irr)
kappa2(d[, c("raterA", "raterB")]) # Cohen
kappa2(d[, c("raterA", "raterB")], weight = "squared") # quadratic
kappa2(d[, c("raterA", "raterB")], weight = "equal") # linear
kappam.fleiss(d[, 2:5]) # Fleiss, 4 raters
icc(d[, 2:4], model = "twoway", type = "agreement", unit = "single")

In icc(), unit = "single" gives ICC(2,1) and unit = "average" gives ICC(2,k). Use type = "agreement" rather than "consistency" unless systematic rater differences are genuinely irrelevant to you, because consistency ignores them.

For the prevalence and bias indices and PABAK:

library(epiR); epi.kappa(table(d$raterA, d$raterB))

SPSS

Cohen's kappa is in Crosstabs. The two rater variables must be coded identically and the table must be square.

CROSSTABS TABLES = raterA BY raterB / STATISTICS = KAPPA.

SPSS gives unweighted kappa only. Weighted kappa requires a macro or the extension command; most people compute it in R instead.

For the ICC:

RELIABILITY VARIABLES = rater1 rater2 rater3 / ICC = MODEL(RANDOM) TYPE(ABSOLUTE) CIN=95.

MODEL(RANDOM) is the two-way random model and TYPE(ABSOLUTE) is absolute agreement. The output table has two rows, "Single Measures" and "Average Measures". Read the one matching how the instrument will actually be used.

Stata

kap raterA raterB
kap raterA raterB, wgt(w)
kappa rater1 rater2 rater3 rater4
icc score subject rater

kap is for two raters, kappa for more. The wgt(w) option takes w for linear or w2 for quadratic weights. icc reports both individual and average forms with confidence intervals, and you specify , mixed if the raters are the entire population of interest.

Check the table is square before you trust kappa

If one rater never used a category the other used, software may silently drop it or return an error that looks unrelated. Cross-tabulate the two raters and look at the table before computing anything. An empty row or column is itself a finding about the coding scheme.

Avoid these

Six mistakes that cost marks

MistakeWhy it matters
Reporting percentage agreement aloneTwo raters with no shared judgement agree 82% of the time when the outcome is 10% prevalent.
Using Pearson's r for agreementA rater scoring three points high every time gives r = 1.00 and zero agreement.
Unweighted kappa on ordered categoriesTreats mild against severe as the same error as mild against moderate.
Not naming the weighting schemeThe same table gave 0.484, 0.541 and 0.609. The number alone is uninterpretable.
Reporting ICC(2,k) for single-rater use0.950 against 0.863 on the same data. Averaging raters cancels error one rater cannot.
Reporting kappa with no intervalA kappa of 0.680 on 100 cases has a 95% interval of 0.536 to 0.824.

A seventh, which is less a statistical error than a procedural one: computing reliability after the raters have discussed the cases. Once they have conferred, independence is gone and the coefficient measures the conversation. Code independently, compute the coefficient, then meet to resolve disagreements, and report the figure from before the meeting.

High agreement but a disappointing kappa?

When one category dominates, two raters can agree on 90% of cases and still score 0.44, and that is the statistic behaving correctly. Send the ratings and you get the right coefficient with its interval, plus the prevalence and bias indices that explain it.

See SPSS data analysis
Tara SutariaTara SutariaSenior Statistical Analyst

Writing it up

How to report it

✓Which coefficient, and why that one for this data.
✓The number of raters, the number of cases, and whether raters were independent.
✓The coefficient with a 95% confidence interval.
✓Observed agreement alongside it, so a reader can see the prevalence effect.
✓For weighted kappa, the weighting scheme by name.
✓For an ICC, the model and whether single or average measures.
✓How disagreements were resolved, and at what point in the process.

Two sentences that do all of it:

"Two reviewers independently screened 100 abstracts. Agreement was 84% (Cohen's κ = 0.68, 95% CI 0.54 to 0.82); the 16 discordant records were resolved by discussion after the coefficient was computed."

"Three assessors rated 8 participants. The two-way random-effects ICC for absolute agreement was 0.86 (single measures), indicating acceptable reliability for use by a single assessor."

Common questions

Frequently asked questions

What is a good inter-rater reliability score?

By the Landis and Koch convention, above 0.80 is almost perfect, 0.61 to 0.80 substantial and 0.41 to 0.60 moderate. Those labels were described as arbitrary by the authors who proposed them. What is acceptable depends on the stakes: 0.65 is reassuring for exploratory coding of interviews and inadequate for a diagnostic decision that determines treatment.

Why is my kappa low when agreement is high?

Almost always because one category dominates. With 90% of cases in one category, two raters acting independently would agree 82% of the time, so 90% observed agreement leaves only a small margin above chance and kappa comes out at 0.444. Report the prevalence index alongside kappa and state agreement on the rare category separately.

Should I use weighted or unweighted kappa?

Weighted whenever the categories are ordered, because a one-step disagreement is genuinely less serious than a two-step one. Name the scheme: on the same table, unweighted gave 0.484, linear weights 0.541 and quadratic weights 0.609. Quadratic is conventional in imaging, linear is more conservative, and the choice should be made before seeing the result.

Can I use Cohen's kappa for three raters?

No. Cohen's kappa is defined for two. Use Fleiss' kappa for three or more, which also allows different raters to assess different subjects. Pairwise Cohen's kappas are still useful as a diagnostic, since they identify which particular rater is out of step in a way that Fleiss' single summary cannot.

Which ICC should I report?

Match it to use. Report single measures if one rater will score each case in practice, and average measures only if the mean of several raters is the score that gets used. On the same worked data these were 0.863 and 0.950. Use the two-way random model with absolute agreement unless systematic differences between raters genuinely do not matter to you.

Is Krippendorff's alpha better than kappa?

It is more flexible rather than better. It handles missing data, any number of raters and nominal, ordinal or interval levels within one framework, which makes it popular in content analysis. For two raters and complete nominal data it gives essentially the same answer as Cohen's kappa, so there is no advantage in switching for its own sake.

How many cases do I need for a reliability study?

More than most studies use. The interval on a kappa of 0.68 from 100 cases runs from 0.54 to 0.82, so 30 cases produces an interval too wide to support any claim about which benchmark band you are in. Around 100 is a reasonable working minimum for a kappa you intend to interpret, and if the category is rare, deliberately over-sample positive cases.

Can I use Cronbach's alpha for inter-rater reliability?

It is the wrong tool. Alpha measures internal consistency among items within a scale, not agreement between raters about the same object. It is also insensitive to systematic differences, so two raters consistently a point apart can produce a high alpha. Use kappa for categories and the ICC for continuous ratings.