Guides / ROC and AUC
ROC curves and AUC
An ROC curve shows how a test or model trades false alarms against missed cases as you move the threshold, and the area beneath it compresses that into one number. It is a good measure of ranking and a poor measure of usefulness, which is why a model with an excellent AUC can still be unusable in practice. This guide covers both, and the calibration question AUC cannot answer.
Written and reviewed by Sidra Khan, Lead BiostatisticianAn ROC curve plots sensitivity against one minus specificity at every possible threshold, tracing the trade-off between catching cases and raising false alarms. The area under it, the AUC, has a direct interpretation: it is the probability that a randomly chosen person with the condition scores higher than a randomly chosen person without it. An AUC of 0.5 is a coin toss and 1.0 is perfect separation. It measures ranking, not whether the predicted probabilities are correct.
The plot
What the curve shows
Any test producing a number needs a cut-off to produce a decision. Each possible cut-off gives a pair of values: a sensitivity and a specificity. The ROC curve is simply all of those pairs plotted at once, sensitivity up the side and one minus specificity along the bottom.
The diagonal is what you would get by guessing. A curve bowing towards the top-left corner means the test separates the two groups well, because you can achieve high sensitivity without paying much in false positives. A curve lying on the diagonal means the score carries no information about who has the condition.
Because it contains every threshold at once, the ROC curve describes the test rather than any particular decision rule. That is its strength when comparing tests and its weakness when deciding what to do with one, since in practice you have to choose a single cut-off and live with it.
The number
Reading the AUC
The most useful interpretation is the ranking one: take one person with the condition and one without, at random. The AUC is the probability the first scores higher. An AUC of 0.80 means that happens 80% of the time.
| AUC | Conventional reading |
|---|---|
| 0.5 | No discrimination at all |
| 0.5 to 0.7 | Poor |
| 0.7 to 0.8 | Acceptable |
| 0.8 to 0.9 | Excellent |
| Above 0.9 | Outstanding, and worth checking for overfitting or leakage |
These bands are conventions, not thresholds with any theoretical standing. An AUC of 0.65 can be valuable for a cheap screening step and an AUC of 0.92 can be useless if it was measured on the data the model was built from.
An AUC calculated on the same data used to fit the model is inflated, often substantially, and the smaller the dataset the worse it gets. It needs internal validation by bootstrapping or cross-validation at minimum, and external validation on genuinely new data before anyone should rely on it.
Worked example one
Building a curve and its AUC by hand
Ten patients are scored by a model. Five have the condition, five do not.
scores, condition present: 8, 7, 6, 5, 3
scores, condition absent: 6, 4, 3, 2, 1
Step 1. Work out sensitivity and specificity at every threshold
Take each distinct score as a cut-off and count how many of each group fall at or above it.
| Threshold | TP | FP | Sensitivity | 1 − specificity |
|---|---|---|---|---|
| ≥ 8 | 1 | 0 | 0.20 | 0.00 |
| ≥ 7 | 2 | 0 | 0.40 | 0.00 |
| ≥ 6 | 3 | 1 | 0.60 | 0.20 |
| ≥ 5 | 4 | 1 | 0.80 | 0.20 |
| ≥ 4 | 4 | 2 | 0.80 | 0.40 |
| ≥ 3 | 5 | 3 | 1.00 | 0.60 |
| ≥ 2 | 5 | 4 | 1.00 | 0.80 |
| ≥ 1 | 5 | 5 | 1.00 | 1.00 |
Plot the last two columns against each other and you have the ROC curve. Each row is one point, and the steps between them are what gives the curve its staircase shape in small samples.
Step 2. Area under the curve, by trapezoids
Join the points and sum the areas beneath. Working left to right across the plot gives AUC = 0.840.
Step 3. The same answer, a completely different way
AUC equals the proportion of case-control pairs in which the case scores higher. There are 5 × 5 = 25 such pairs. Count them, giving half credit for ties:
case 8 beats all 5 → 5.0
case 7 beats all 5 → 5.0
case 6 beats 4, ties with one 6 → 4.5
case 5 beats 4 → 4.0
case 3 beats 2, ties with one 3 → 2.5
total = 21 of 25 → AUC = 0.840
The two methods agree exactly, and that is not a coincidence: the AUC is the Mann-Whitney U statistic rescaled. It means AUC is a rank-based measure. It depends only on the order of the scores and not on their values, which is precisely why it can be high while the predicted probabilities themselves are badly wrong.
Worked example two
A model with excellent AUC and useless probabilities
A risk model is validated on a new population. Patients are grouped into five bands by predicted risk, and the observed event rate in each band is compared with what the model predicted.
| Risk band | Model predicted | Actually observed |
|---|---|---|
| 1 (lowest) | 10% | 5% |
| 2 | 20% | 10% |
| 3 | 40% | 20% |
| 4 | 60% | 30% |
| 5 (highest) | 100% | 50% |
What the AUC says
The ordering is perfect. Band 5 really is highest risk, band 1 really is lowest, and every band sits where it should. Because AUC depends only on ranking, it is excellent and completely unchanged by anything in the second column.
What the calibration plot says
Every prediction is twice the truth. Plotted as predicted against observed, the line sits far above the diagonal across the whole range. A clinician told a patient has a 60% risk is working from a number that should be 30%.
Why this is not a hypothetical
It is the normal consequence of applying a model built in one population to another with different underlying risk. Discrimination transfers reasonably well between settings, because ranking is fairly robust. Calibration does not, because it depends on the baseline rate, in the same way predictive values depend on prevalence while sensitivity and specificity do not.
A model being validated externally should be recalibrated to the new population, and the paper should report calibration before and after. Reporting AUC alone hides the entire problem.
Decisions
Picking a threshold from it
Youden's J, sensitivity plus specificity minus one, identifies the point furthest from the diagonal and is the usual default. It weights a missed case and a false alarm equally, which almost no clinical situation does.
Honesty
Four things AUC does not tell you
| Limitation | Why it matters |
|---|---|
| It ignores calibration | A model can rank people perfectly while every predicted probability is wrong by a factor of three. AUC would not change. |
| It weights all errors equally | The whole curve is summarised, including regions of the threshold space nobody would ever operate in. |
| It hides prevalence | AUC is unaffected by how common the condition is. Positive predictive value is not, so a strong AUC can still mean most positives are false. |
| It flatters imbalanced data | With a rare outcome, large numbers of true negatives keep the false positive rate low and the curve looks good. A precision-recall curve exposes what ROC hides. |
The last two travel together and explain most disappointments when a promising model meets a real population. If your outcome occurs in a few per cent of people, report a precision-recall curve alongside the ROC.
The alternative
Precision-recall curves, and when to use them
When the outcome is rare, an ROC curve can look excellent while the model is unusable. A precision-recall curve shows what ROC hides.
The two plots, side by side
An ROC curve plots sensitivity against one minus specificity. A precision-recall curve plots the same sensitivity, called recall here, against precision, which is the same quantity as positive predictive value. The difference is the denominator. Specificity is measured against the true negatives, of which there are a great many when the outcome is rare. Precision is measured against everything the model flagged, which is what a user actually experiences.
A worked comparison at 1% prevalence
Take 10,000 people, 100 of whom have the condition.
| Operating point | TP | FP | ROC point | Precision | Flagged per true case |
|---|---|---|---|---|---|
| Sens 80%, spec 90% | 80 | 990 | (0.10, 0.80) | 7.5% | 13.4 |
| Sens 80%, spec 99% | 80 | 99 | (0.01, 0.80) | 44.7% | 2.2 |
| Sens 50%, spec 99.9% | 50 | 10 | (0.001, 0.50) | 83.5% | 1.2 |
The first row sits close to the top-left corner of an ROC plot and would be described as strong performance. It also flags 1,070 people to find 80 cases, so fewer than one in thirteen of those flagged has the condition. Nothing in the ROC curve communicates that.
The second row looks only marginally better on ROC, moving from 0.10 to 0.01 on an axis that is hard to read at that scale. In precision it is six times better. The improvement that matters is almost invisible on the plot most papers show.
Reading a precision-recall curve
If the outcome occurs in fewer than roughly one in ten, show a precision-recall curve as well. The ROC curve answers whether the model can rank people. The precision-recall curve answers whether anyone could act on it.
The other half
Discrimination is not calibration
Discrimination is whether the model puts the right people in the right order. Calibration is whether the numbers it produces are true. AUC measures only the first, and a model used to inform anyone needs both.
If a model says 100 patients each have a 20% risk, roughly 20 of them should have the event. If 45 do, the model is badly calibrated even with a respectable AUC, and acting on its stated probabilities would be a mistake.
Model selection
Comparing two models
Comparing AUCs computed on the same patients means comparing correlated quantities, so an ordinary test is not appropriate. DeLong's test accounts for that correlation and is the standard approach.
A statistically significant AUC improvement can still be clinically trivial. Adding a variable that lifts AUC from 0.78 to 0.80 may change no decision for any patient. Decision curve analysis addresses this directly by asking whether using the model leads to better decisions than treating everyone or nobody, across a range of threshold probabilities.
Doing it
In R, SPSS and Stata
R
pROC is the standard package and handles curves, intervals, thresholds and comparisons.
library(pROC)
r <- roc(outcome ~ score, data = df)
auc(r); ci.auc(r)
coords(r, "best", best.method = "youden")
Compare two models on the same patients with roc.test(r1, r2, method = "delong"). For calibration, rms::val.prob(predicted, observed) returns the calibration slope, intercept and Brier score together with a plot, and rms::calibrate() handles bootstrap-corrected calibration for a model you fitted yourself.
SPSS
ROC analysis is under Analyze, Classify, ROC Curve.
ROC score BY outcome (1)
/PLOT = CURVE(REFERENCE)
/PRINT = SE COORDINATES
/CRITERIA = CUTOFF(INCLUDE) TESTPOS(LARGE) DISTRIBUTION(FREE) CI(95).
The coordinate table gives sensitivity and one minus specificity at every cut-off, so specificity is one minus the second column and Youden's J can be computed from it. SPSS produces no calibration output, so group the predicted probabilities and compare with observed rates manually.
Stata
roctab outcome score, detail graph
That returns the AUC with an interval and the full coordinate table. Compare two correlated AUCs with roccomp outcome score1 score2, graph, which uses the DeLong approach. After a logistic model, lroc plots the curve directly and estat gof, group(10) table gives the Hosmer-Lemeshow grouping, which is worth reading as a table rather than as a p-value.
Every command above will happily compute an AUC on the data the model was fitted to, and it will be optimistic, often by a wide margin in small samples. Use bootstrap optimism correction, cross-validation, or an external dataset, and say in the paper which one the reported figure came from.
Avoid these
Six mistakes that cost marks
| Mistake | Why it matters |
|---|---|
| Reporting AUC with no calibration | A model can rank perfectly while every probability is twice what it should be. AUC would not change. |
| Quoting an AUC from the training data | Optimistic, sometimes dramatically. Internal validation at minimum, external by preference, and say which. |
| Using Youden by default | It weights a missed case and a false alarm equally, which almost no clinical situation does. |
| Comparing AUCs with an ordinary test | Two AUCs on the same patients are correlated. Use DeLong, and report the difference with an interval. |
| Treating AUC as clinical usefulness | A rise from 0.78 to 0.80 may change no decision for anyone. Decision curve analysis asks the question AUC cannot. |
| ROC alone on heavily imbalanced data | Abundant true negatives hold the false positive rate down and flatter the curve. Add a precision-recall curve. |
Writing it up
How to report it
TRIPOD is the reporting standard for prediction model studies and most journals expect it, as STROBE is expected for observational work. If a model is already built and you want it checked before publication, that is what an independent statistical review covers.
Common questions
Frequently asked questions
What does AUC mean?
The probability that a randomly chosen person with the condition receives a higher score than a randomly chosen person without it. An AUC of 0.80 means that happens 80% of the time. It measures how well the model ranks people, not whether its predicted probabilities are accurate.
What is a good AUC?
Conventionally 0.7 to 0.8 is acceptable, 0.8 to 0.9 excellent and above 0.9 outstanding, but these are conventions rather than standards. Context decides: a modest AUC can be valuable in a cheap triage step, and a very high one measured on the data the model was fitted to may mean nothing at all.
What is the difference between discrimination and calibration?
Discrimination is whether the model puts people in the right order, which is what AUC measures. Calibration is whether the predicted probabilities are correct in absolute terms. A model can rank perfectly while systematically overstating everyone's risk, so a prediction model needs both reported.
Should I use Youden's index to choose a threshold?
Only if a missed case and a false alarm are genuinely equally bad, which is rare. Youden weights both errors equally by construction. It is usually better to state the clinical requirement, for example the threshold achieving 95% sensitivity, and report the specificity that follows.
Is AUC affected by prevalence?
No, which is both its strength and its limitation. It stays stable across populations with different prevalence, so it transfers well. It therefore says nothing about how many positives will be false, which does depend on prevalence and is what users of the test care about.
How do I compare two AUCs?
With DeLong's test when both are computed on the same patients, because the two estimates are correlated and an ordinary comparison would be wrong. Report the difference with a confidence interval, and consider whether the improvement is large enough to change any decision.
Keep reading