Plotwright

Worked diagnostic-accuracy example

An ROC AUC is a ranking result—not a diagnostic cutoff.

AUC combines performance across every threshold. A real classification decision still requires one threshold, its sensitivity and specificity, uncertainty, and an explicit judgment about false positives and false negatives.

109 CT images5 rating levelsAUC + 95% CI9-minute read

The short answer

Use AUC to describe discrimination; use a threshold to describe decisions.

An ROC curve plots sensitivity against the false-positive rate (1 − specificity) while sweeping the threshold from most to least stringent. AUC summarizes how well higher measurements rank condition-positive observations above condition-negative observations across that entire sweep.

Hanley–McNeil interpretation: AUC is the probability that a randomly chosen positive observation receives a higher rating than a randomly chosen negative observation, with ties contributing half. It does not itself choose a clinically useful cutoff.

The canonical 109-image rating example

Hanley and McNeil's 1982 worked example contains 58 normal and 51 abnormal CT images, each assigned an ordinal suspicion rating from 1 to 5. Higher values indicate greater suspicion of abnormality.

True status Rating 1 Rating 2 Rating 3 Rating 4 Rating 5 Total
Normal 33 6 6 11 2 58
Abnormal 3 2 2 11 33 51
ROC curve for the Hanley–McNeil CT rating data Six ROC points form a curve with area 0.893. The cutoff at rating four or greater has 86.3 percent sensitivity and 77.6 percent specificity. Predict none positive Rating ≥5: sensitivity 64.7%, specificity 96.6% Rating ≥4: sensitivity 86.3%, specificity 77.6% Rating ≥3: sensitivity 90.2%, specificity 67.2% Rating ≥2: sensitivity 94.1%, specificity 56.9% Rating ≥1: predict all positive 0 0.5 1.0 0 0.5 1.0 False-positive rate (1 − specificity) Sensitivity Youden cutoff: rating ≥4 AUC = 0.893
Each point is one rating threshold. Moving toward the upper right detects more abnormal images while also classifying more normal images as positive.

Exact Plotwright results

Discrimination across thresholds AUC = 0.893

SE 0.0325; Hanley–McNeil 95% CI 0.829 to 0.957; n = 109.

Youden-selected threshold Rating ≥4

Sensitivity 86.3%; specificity 77.6%; Youden J = 0.639.

At rating ≥4, the sensitivity 95% CI is 74.3% to 93.2% and the specificity 95% CI is 65.3% to 86.4% (Wilson intervals). The positive likelihood ratio is 3.85 and the negative likelihood ratio is 0.177. These intervals matter: the same point estimates would be less convincing in a smaller study.

One curve, several legitimate operating points

Positive when rating is Sensitivity Specificity Practical tradeoff
≥5 64.7% 96.6% Few false positives; more abnormal images missed.
≥4 86.3% 77.6% Maximum Youden J for this dataset.
≥3 90.2% 67.2% More sensitivity at the cost of specificity.
≥2 94.1% 56.9% Detects most abnormal images; many false positives.
Youden is not a clinical policy: maximizing sensitivity + specificity − 1 weights the two error types equally and ignores downstream consequences. A screening workflow, confirmatory test and scarce invasive procedure can rationally choose different cutoffs from the same ROC curve.

Six checks before reporting an ROC result

  1. Name the positive condition and score direction. Reversing them changes AUC to 1 − AUC.
  2. Define the reference standard. “Normal” and “abnormal” require an independently justified status definition.
  3. Describe the intended-use population. Accuracy can change with case spectrum, setting and selection.
  4. Report uncertainty. Give the AUC confidence interval and sensitivity/specificity intervals at the chosen cutoff.
  5. Prespecify or validate the cutoff. Selecting and evaluating a threshold on the same sample is optimistic.
  6. Connect errors to consequences. AUC does not encode prevalence, predictive values, costs, benefits or clinical utility.

What AUC cannot establish

A large AUC shows useful ranking in the studied sample. It does not prove that a selected cutoff transports to another population, that predicted probabilities are calibrated, that the reference standard is unbiased, or that acting on the test improves outcomes. A tiny P value against AUC = 0.5 answers only a no-discrimination null; it is not a certificate of clinical usefulness.

The five-point ratings in this example are ordinal judgments from a historical CT reading study, not calibrated probabilities or validation of a contemporary model. The example is valuable because the complete rating counts and published AUC are reproducible—not because its operating point should be reused.

A compact reporting template

“The ordinal rating discriminated 51 abnormal from 58 normal images with AUC 0.893 (95% CI 0.829–0.957). Higher ratings indicated abnormality. At the prespecified positive threshold of rating ≥4, sensitivity was 86.3% (95% CI 74.3%–93.2%) and specificity was 77.6% (95% CI 65.3%–86.4%). The reference standard, intended-use population and handling of indeterminate results were defined before analysis.”

This template demonstrates the quantities, not the original study's full reporting. Replace the threshold-selection statement, population, reference standard and every estimate with the actual study design and results.

What Plotwright checks

Plotwright expands the published rating counts into all 109 observations and runs them through the production ROC engine. The release gate expects AUC 0.893 within 0.001. Separate tests verify the entire curve against the rank-based identity AUC = Mann–Whitney U / (npositive × nnegative), the Hanley–McNeil standard error, Wilson intervals, likelihood ratios and cutoff scan.

Those checks support defined numerical outputs. They cannot establish the validity of a reference standard, prevent spectrum bias, choose the consequences worth optimizing or make a diagnostic decision for a researcher.

Sources and reproducibility

  1. Hanley & McNeil (1982), Radiology—the original AUC interpretation, uncertainty method and 109-image example.
  2. FDA statistical guidance for diagnostic-test studies—study context, reference standards, paired accuracy measures and confidence intervals.
  3. STARD 2015 reporting guideline—essential reporting items for diagnostic-accuracy studies.
  4. Plotwright statistical validation—the published benchmark, numerical tolerance and evidence boundaries.

Keep the curve, cutoff table and uncertainty together.

Plotwright turns two columns of reference-negative and reference-positive values into an inspectable ROC curve, AUC interval and every sensitivity/specificity pair.

Open the exact ROC sample Read: Kaplan-Meier curves