Sensitivity, specificity, PPV and NPV

Everything comes from four counts: how the test result compares with the truth, established by a reference standard.

Read down the columns and you get sensitivity and specificity. Read across the rows and you get the predictive values. That is the whole difference between the two pairs, and it is why they behave differently.

Sensitivity = TP / (TP + FN). Of everyone who has the condition, the share the test catches. A sensitive test rarely misses.

Specificity = TN / (TN + FP). Of everyone without it, the share the test clears. A specific test rarely raises a false alarm.

There is a trade-off between them whenever the test produces a number rather than a yes or no. Lower the threshold and you catch more cases while creating more false alarms. The two move in opposite directions, so a test is never simply "better" without saying at what threshold and for what purpose.

PPV = TP / (TP + FP). Given a positive result, the chance the condition is really there.

NPV = TN / (TN + FN). Given a negative result, the chance it really is not.

These are the numbers a patient actually cares about, and they depend on prevalence. The test has not changed; the mix of people being tested has. In a population where almost nobody has the condition, nearly everyone the test flags will be a false alarm, simply because there are so many more well people available to be flagged.

A new test is evaluated against a reference standard in 420 patients. Sixty turn out to have the condition. Here is the table, and then every measure derived from it.

Step 1. Sensitivity

Read down the first column. Of the 60 people who have the condition, 54 tested positive.

sensitivity = 54 / 60 = 0.900, so 90.0%, 95% CI 79.9% to 95.3%.

Note how wide that interval is. Sensitivity rests on only 60 people, because the 360 without the condition contribute nothing to it. A study can have 420 participants and still estimate sensitivity poorly.

Step 2. Specificity

Read down the second column. Of 360 people without the condition, 324 tested negative.

specificity = 324 / 360 = 0.900, so 90.0%, 95% CI 86.5% to 92.7%.

Same point estimate as sensitivity, much tighter interval, because six times as many people inform it.

Step 3. Positive predictive value

Now read across the top row. Of the 90 people the test flagged, 54 really had the condition.

PPV = 54 / 90 = 0.600, so 60.0%, 95% CI 49.7% to 69.5%.

A positive result on a test with 90% sensitivity and 90% specificity is right 60% of the time here. That is not a flaw in the test. It is what those numbers produce at this prevalence.

Step 4. Negative predictive value

Read across the bottom row. Of 330 negatives, 324 were genuinely clear.

NPV = 324 / 330 = 0.982, so 98.2%, 95% CI 96.1% to 99.2%.

Step 5. Likelihood ratios

LR+ = sensitivity / (1 − specificity) = 0.90 / 0.10 = 9.0

LR− = (1 − sensitivity) / specificity = 0.10 / 0.90 = 0.111

A positive result makes the condition nine times more likely than it was. A negative result cuts the odds to about a ninth.

Step 6. Prevalence, accuracy and Youden's J

prevalence = 60 / 420 = 14.3%  ·  accuracy = (54 + 324) / 420 = 90.0%  ·  J = 0.90 + 0.90 − 1 = 0.80

Use the Wilson interval rather than the ordinary normal approximation for all of these. Sensitivity and specificity often sit near 90% or above, where the normal approximation produces intervals that extend past 100%.

The test above now goes into a screening programme where 2% of those attending have the condition. Nothing about the test changes: sensitivity stays at 90%, specificity at 90%. Only the mix of people changes.

Step 1. Rebuild the table at the new prevalence

Take 10,000 attendees. At 2% prevalence, 200 have the condition and 9,800 do not.

TP = 200 × 0.90 = 180  ·  FN = 20  ·  TN = 9,800 × 0.90 = 8,820  ·  FP = 980

Step 2. Recalculate the predictive values

PPV = 180 / (180 + 980) = 180 / 1,160 = 15.5%

NPV = 8,820 / (8,820 + 20) = 99.8%

Step 3. Translate it into something a clinician would recognise

1,160 people are told their result is positive. 180 of them have the condition. That is 6.4 positive results for every true case, and 980 people sent for further investigation who did not need it.

The first three rows are identical. The fourth has fallen by three quarters. Any paper quoting sensitivity, specificity and accuracy alone would describe these two situations with the same numbers.

Likelihood ratios solve the problem that predictive values are population-specific while sensitivity and specificity do not tell you what a result means.

LR+ = sensitivity / (1 − specificity). How much more likely a positive result is in someone with the condition than someone without.

LR− = (1 − sensitivity) / specificity. The same for a negative result.

They are properties of the test, so they travel between populations, and they combine with prevalence through Bayes' rule: pre-test odds multiplied by the likelihood ratio gives post-test odds. For the test above, LR+ is 9.5 and LR− is 0.056. At 1% prevalence the pre-test odds are 0.0101, so the post-test odds after a positive are 0.096, which is a probability of 8.8%. Exactly the figure the 2 by 2 table gave.

Accuracy is (TP + TN) / total, and it is the most quoted and least useful figure in diagnostics.

Consider a condition affecting 1% of people and a "test" that always returns negative. It never catches a single case. Its sensitivity is zero. Its accuracy is 99%, and it would beat most real tests on that measure alone.

This is the same problem that makes accuracy a poor measure for any classifier on imbalanced data. Report sensitivity and specificity with confidence intervals, give predictive values alongside the prevalence they assume, and treat a headline accuracy figure as a warning sign rather than a result.

When a test yields a continuous measurement, the cut-off is a choice, and an ROC curve shows every option at once. It and it should follow from consequences rather than from whatever maximises a statistic.

R

The epiR package produces everything above from a 2 by 2 matrix, with exact confidence intervals, in one call. Enter the counts in the order TP, FP, FN, TN, filling the matrix by row.

library(epiR)
tab <- matrix(c(54, 36, 6, 324), nrow = 2, byrow = TRUE)
epi.tests(tab, conf.level = 0.95)

This returns sensitivity, specificity, both predictive values, both likelihood ratios, prevalence and accuracy, each with an interval. For a model rather than a fixed test, pROC::roc() gives the curve and pROC::coords() the measures at any threshold.

SPSS

SPSS has no single diagnostic accuracy command, so build the table and derive the measures from it.

CROSSTABS TABLES = testresult BY goldstandard
  /CELLS = COUNT
  /STATISTICS = CHISQ.

For a continuous score use ROC VARIABLES = score BY outcome (1) /PLOT = CURVE /PRINT = COORDINATES SE. The coordinate table lists sensitivity and one minus specificity at every cut-off, so specificity is one minus the column shown. Predictive values are not produced and must be calculated from the counts.

Stata

The user-written diagt command is the usual route and gives the full set directly.

ssc install diagt
diagt goldstandard testresult

For immediate counts without a dataset, diagti 54 36 6 324 takes the four cells in the same order. For a continuous score, roctab outcome score, detail gives the curve and the coordinates.

Accuracy estimates are often too optimistic, and usually for one of four reasons.

The first two matter most and are the easiest to spot in a paper. A study with no borderline patients in it is describing a situation where nobody would have ordered the test.

STARD is the reporting standard for diagnostic accuracy studies and most journals expect it, in the same way STROBE is expected for observational work. If the analysis is already drafted and you want it checked before submission, that is what an independent statistical review covers.