The short answer
Start with the design; inspect expected counts.
For independent categorical observations, Pearson's chi-square tests association by comparing observed and expected counts. Its P value is asymptotic. Fisher's exact test uses the hypergeometric distribution for a 2×2 table and remains valid when the chi-square approximation is doubtful.
R's canonical tea-tasting experiment
R's official fisher.test documentation gives the
Fisher tea-drinker example: eight cups, four prepared milk-first
and four tea-first. The taster classified three of each
preparation correctly and one incorrectly.
| Guess | Actually milk first | Actually tea first | Total |
|---|---|---|---|
| Milk first | 3 | 1 | 4 |
| Tea first | 1 | 3 | 4 |
| Total | 4 | 4 | 8 |
What the chi-square approximation sees
Under independence, each expected count is
row total × column total / N = 4×4/8 = 2. All four
expected cells are below 5, so the usual large-sample heuristic
is not satisfied. The uncorrected Pearson statistic is
χ²(1) = 2.000, giving
P = 0.1573 from the chi-square reference
distribution.
Applying Yates' continuity correction changes the statistic to χ²(1) = 0.500 and P = 0.4795. That correction is a separate convention; it does not turn the calculation into Fisher's conditional exact test.
What Fisher's exact calculation sees
With both row and column margins fixed at four, there are C(8,4) = 70 equally likely allocations under the null. The observed table has probability 16/70. The pre-specified positive tail also includes the perfect 4-and-0 table, giving 17/70 = 0.2429, the one-sided result printed in R's example.
Plotwright's linked project uses the default probability-based two-sided definition: add every fixed-margin table whose probability is no greater than the observed table. Here that is 34/70 = 0.4857.
Three P values do not create three scientific answers
Uncorrected chi-square, Yates-corrected chi-square and Fisher's exact test use different reference calculations. Their P values differ sharply in this tiny table because the asymptotic distribution is smooth while the fixed-margin exact distribution has only a few attainable outcomes.
The remedy is not to calculate all three and select the smallest. State the sampling model and inference rule in advance. For a 2×2 table with expected counts of 2 and fixed margins, the conditional exact result is the defensible primary analysis. Other exact unconditional methods may be preferable when margins were not fixed; they are outside this worked example.
Report association, not only a test
The sample cross-product odds ratio is (3×3)/(1×1) = 9.0, but its approximate 95% interval is extremely wide: 0.367 to 220.927. Cramér's V (equal to |φ| in a 2×2 table) is 0.500. These estimates show the apparent direction and the severe imprecision that a binary “significant/not significant” label hides.
The odds ratio reverses to 1/9 if a row or column reference is reversed. Relative risk, risk difference, diagnostic sensitivity and number needed to treat require a design and orientation that make those quantities meaningful. Software displaying them does not make every quantity appropriate for this tea-classification experiment.
A practical 2×2 checklist
- Enter integer counts, not percentages, means or duplicated summary rows.
- Verify that observational units are independent and categories are mutually exclusive.
- Inspect expected counts, not only observed cells or total sample size.
- State whether margins were fixed and whether the alternative was one- or two-sided.
- Report an effect estimate with its orientation and uncertainty.
A compact reporting template
“The association between true and guessed pouring order was tested with a pre-specified two-sided Fisher exact test because the 2×2 table had fixed margins and every expected count was 2. Six of eight classifications were correct (sample odds ratio 9.0, 95% CI 0.37 to 220.93), but the conditional probability-based result was imprecise (P = 0.4857).”
What Plotwright checks
Plotwright submits the four counts to its production contingency engine. The published-example gate independently derives the 70 fixed-margin allocations, the one- and two-sided exact probabilities, the Pearson identity, all expected counts, Yates' result, Cramér's V and the sample odds ratio.
Those checks establish the defined calculations. They cannot establish independence, decide whether margins were fixed, choose tails, or determine which effect measure is scientifically meaningful.
Sources and reproducibility
- R stats documentation: fisher.test—the complete tea-tasting table, fixed-margin null, hypergeometric calculation, directional alternative and probability-based two-sided definition.
- R stats documentation: chisq.test—Pearson chi-square and continuity-correction conventions.
- NIST/SEMATECH: Fisher exact test—sampling models, fixed margins, hypergeometric logic and chi-square as a large-sample approximation.
- NIST/SEMATECH: contingency tables—expected-count and Pearson-statistic identities.
- Plotwright statistical validation—published expected values, tolerances and evidence limits.