The problem
r² answers a question you did not ask.
r² reports how much of the variation in signal the line accounts for. It does not report whether a line is the right shape. A saturating assay will happily return a high r² across its entire range while every interpolated concentration at the top and bottom is biased in a predictable direction.
The calibration series
A dilution series of known concentrations read as absorbance. All seven standards are shown; a calibration you cannot audit is not a calibration.
| Concentration (ng/mL) | 0 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|
| Absorbance | 0.04 | 0.21 | 0.39 | 0.72 | 1.28 | 2.05 | 2.71 |
Doubling the concentration from 2 to 4 ng/mL nearly doubles the signal (0.21 → 0.39). Doubling from 32 to 64 adds about a third (2.05 → 2.71). The assay is flattening out, and the top standards are the ones that carry the most leverage in a straight-line fit.
The same data, two fitted ranges
| Fit | Slope | Intercept | r² |
|---|---|---|---|
| All seven standards (0–64) | 0.041999 | 0.301165 | 0.926297 |
| Linear range only (0–16) | 0.077375 | 0.063750 | 0.997109 |
The two slopes differ by nearly a factor of two. An unknown read against the wrong one is not slightly off; it is off by that factor across most of the working range.
The residuals give it away
Residuals from the seven-point fit, in concentration order:
| Concentration | 0 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|
| Residual (all seven) | −0.2612 | −0.1752 | −0.0792 | +0.0828 | +0.3069 | +0.4049 | −0.2791 |
| Residual (0–16 fit) | −0.0237 | −0.0085 | +0.0168 | +0.0373 | −0.0217 | — | — |
The seven-point residuals are not scattered around zero. They run negative, swing positive through the middle, then collapse at the top standard — the signature of fitting a straight line to a curve. Over 0–16 the residuals are an order of magnitude smaller and change sign without a systematic sweep.
Interpolating an unknown
An unknown reading an absorbance of 0.950 falls inside the linear range. Inverting the 0–16 fit gives (0.950 − 0.063750) / 0.077375 = 11.454 ng/mL. Read against the seven-point fit instead, the same absorbance would return about 15.4 ng/mL — a difference driven entirely by which standards were allowed into the line.
When to fit the curve instead
Restricting to a linear range discards real standards. The alternative is to model the shape the assay actually has: a four-parameter logistic captures the lower and upper asymptotes and lets every standard contribute, at the cost of estimating four parameters instead of two. That is the same machinery used for dose–response fitting — see IC50 and EC50 dose response. With seven standards and no replicates, a four-parameter fit is poorly determined; more standards, replicated, are the fix.
A defensible calibration workflow
- Plot signal against concentration before fitting anything, on the axis scale the assay is designed for.
- Decide the working range from the residual pattern and the assay's known behaviour, not from r².
- State the range in the method, and treat readings outside it as requiring dilution and a re-read.
- Replicate standards so the calibration itself carries uncertainty; a single reading per standard gives an interpolated concentration no error bar.
- Report the fitted parameters and the accepted range alongside the interpolated results.
What Plotwright checks
Both fits above come from the same linear-regression route the published-example suite exercises, and the engine's regression and curve-fitting routes carry reference checks on the validation page. Every number on this page is reproducible from the seven visible standards in the linked editable project.
Sources and reproducibility
- Findlay, J. W. A. & Dillard, R. F. (2007), “Appropriate calibration curve fitting in ligand binding assays”, The AAPS Journal—why calibration-model choice and working range are validated properties of an assay rather than curve-fitting preferences.
- Anscombe's quartet—the general case: identical summary statistics, different data.
- IC50 and EC50 dose response—four-parameter logistic fitting when the linear range is not enough.
- Plotwright statistical validation—published expected values, numerical tolerances and evidence limits.