Choosing a statistical test: a decision table with the scipy call

A student once asked why we had used a t-test. The honest answer was that it was the test we always used. Six treated samples against six controls, log2 intensities, one treated sample well above the rest — and no one had looked at the design before picking the test. “It’s what we always use” is not an answer that survives the next question.

The question stayed with me because the p-value was never the problem. The problem is that a test is a statement about the data: whether the observations are independent, whether the groups are paired, whether the spread is comparable, whether the outcome is a measurement or a count. Get one of those wrong and the p-value is not conservative. It is simply the wrong number.

This is the table I keep next to my keyboard — the question, the test, the assumption that decides the row, and the scipy call. It is not a statistics lesson. It is the checklist I run before I look at a p-value.

The code below needs scipy 1.11 or later — the permutation shortcut needs 1.15. The outputs shown come from scipy 1.15.3 on Python 3.10.10.

Four questions before the test

  1. What kind of outcome is it? A measurement, a count, a category, or a time to an event. This picks the family.
  2. Are the observations independent? Not “did you use a t-test” but: is each row a different patient, or the same patient twice? Six technical replicates of one sample are not six independent observations, whatever the test will do with them.
  3. How many groups, and are they paired? One, two, or more; independent or repeated.
  4. What is the question? A difference in averages, a difference in distributions, an association between two variables, or agreement between two measurements. These are four different questions. A test that answers one of them will still run on data that asks another, which is how most wrong tests happen.

The first three questions choose the row. The fourth decides whether the row’s test answers the question at all.

The table

The question Test The assumption that decides it scipy If it fails
One mean against a fixed value one-sample t values roughly normal (or n large, by the CLT) ttest_1samp(a, mu) Wilcoxon signed-rank
Two independent groups, continuous Student’s t normal-ish in both groups, equal variance ttest_ind(a, b) Welch, then Mann–Whitney
Two paired measurements paired t the paired differences are roughly normal ttest_rel(a, b) Wilcoxon signed-rank
Three or more independent groups one-way ANOVA normal-ish, equal variance f_oneway(*groups) Welch ANOVA, then Kruskal–Wallis
Three or more paired measurements repeated-measures ANOVA sphericity AnovaRM (statsmodels) Friedman
Two categorical variables chi-square expected count of at least 5 in every cell chi2_contingency(table) Fisher exact for 2×2, permutation otherwise
Two continuous variables Pearson linear, bivariate normal, no single point driving it pearsonr(x, y) Spearman, then Kendall
Do two distributions differ? two-sample K–S continuous data, no ties ks_2samp(a, b) Anderson–Darling, permutation
Time to an event log-rank, Cox proportional hazards not in scipy its own post

The “at least 5” for chi-square is Cochran’s guideline, not a theorem: it tells you when the asymptotic p-value stops being trustworthy, which is a reason to reach for Fisher’s exact test, not a reason to collapse categories until the number looks better.

Two boundaries are worth drawing. Time-to-event data gets one row and no more — log-rank and Cox deserve their own post, and the hazard ratio is an effect measure rather than a test. And nothing here covers the case where the unit of analysis is not the unit of observation — pooled samples, nested batches, repeated visits with missing sessions. That is usually a mixed model, not a different t-test.

The same six samples, run properly

The comparison from the opening. Two things to notice before the test: the treated group has a variance four times the control group’s, and the 10.8 sits apart from the rest of its group.

import numpy as np
from scipy import stats

control = np.array([8.2, 8.5, 8.1, 8.7, 8.3, 8.9])
treated = np.array([9.1, 9.6, 9.4, 10.8, 9.2, 9.9])

student = stats.ttest_ind(treated, control)                   # equal_var=True is the default
welch = stats.ttest_ind(treated, control, equal_var=False)

for name, res in [("Student", student), ("Welch", welch)]:
    print(f"{name:8s} t={res.statistic:.2f}  df={res.df:.1f}  p={res.pvalue:.4f}")

Student  t=4.28  df=10.0  p=0.0016
Welch    t=4.28  df=7.3  p=0.0033

With equal group sizes the two tests share the same numerator, and the pooled and unpooled standard errors coincide, so the t-statistic is identical. What differs is the degrees of freedom (10 against 7.3) and, with it, the p-value — by a factor of two. Both are small here. That is luck. With a slightly smaller effect the two numbers would straddle 0.05, and the test, not the biology, would have decided the paper.

ttest_ind defaults to Student. That default is a historical choice, not a recommendation; pass equal_var=False unless you have a reason not to.

The second half of the answer is the size of the difference:

import numpy as np
from scipy import stats

control = np.array([8.2, 8.5, 8.1, 8.7, 8.3, 8.9])
treated = np.array([9.1, 9.6, 9.4, 10.8, 9.2, 9.9])

welch = stats.ttest_ind(treated, control, equal_var=False)
ci = welch.confidence_interval()

n1, n2 = len(treated), len(control)
sp = np.sqrt(((n1 - 1) * treated.var(ddof=1) + (n2 - 1) * control.var(ddof=1)) / (n1 + n2 - 2))
print(f"difference {treated.mean() - control.mean():.2f} log2, "
      f"95% CI [{ci.low:.2f}, {ci.high:.2f}], d={(treated.mean() - control.mean()) / sp:.2f}")

difference 1.22 log2, 95% CI [0.55, 1.88], d=2.47

The difference is 1.22 log2 units, somewhere between 1.5-fold and 3.7-fold. Cohen’s d of 2.5 sounds enormous, and the interval still lets the effect range over a factor of two and a half. That is not a flaw in the test. It is six samples per group, and no test repairs it.

So the methods sentence is not “a t-test was used”. It is: groups compared with Welch’s t-test; mean difference reported with a 95% confidence interval; n per group, with biological replicates as the unit.

When an assumption fails

In the order I try things:

  1. Change the test, keep the question. Welch instead of Student costs nothing and is right more often. A rank test is a different question — it compares typical values, not means — so use one when that is what you actually want to know, not as a punishment for non-normality.
  2. Keep the test, change the inference. A permutation test asks how often the observed statistic arises when the group labels carry no information.
  3. Change the question. If the design cannot answer it — three samples, confounded batches, no control — no test will. That is a result about the design, and it is worth reporting as one.

The permutation shortcut for the t-test is one line:

# continuing from the example above
stats.ttest_ind(treated, control, method=stats.PermutationMethod(n_resamples=10_000))

It assumes exchangeability under the null, which is a real assumption. It is not permission to ignore pairing or batch structure.

The same four numbers, two questions

A diagnostic test, 120 samples, 40 with the condition and 80 without. The test is positive in 34 of the 40 and in 10 of the 80.

import numpy as np
from scipy import stats

table = np.array([[34, 10],
                  [6, 70]])           # rows: test positive/negative, columns: condition present/absent

res = stats.chi2_contingency(table)
print(f"chi2={res.statistic:.1f}  p={res.pvalue:.1e}  min expected={res.expected_freq.min():.1f}")
print(f"sens={34/40:.0%}  spec={70/80:.0%}  PPV={34/44:.0%}  NPV={70/76:.0%}")

chi2=57.3  p=3.8e-14  min expected=14.7
sens=85%  spec=88%  PPV=77%  NPV=92%

The chi-square answers one question: are the rows associated with the columns? Here, emphatically yes. It says nothing about whether the test is any good. Association is cheap at n=120 — a test barely better than a coin will still produce a small p-value — and the p-value does not tell you how much better the test is.

Sensitivity and specificity belong to the test. The predictive values belong to the test in a population — they move when the prevalence moves, and sensitivity and specificity do not. At 2% prevalence, the same 85% and 88% give

PPV = 0.85 × 0.02 / (0.85 × 0.02 + 0.125 × 0.98) = 0.12

Twelve per cent. One positive in eight is real. The test has not changed; the prevalence has, and the number a patient cares about fell from 77% to 12%. Same test, different waiting room, a different answer.

Two traps

Testing normality to choose the test. Shapiro–Wilk and its relatives answer “is there evidence of non-normality”, which is not the question “is it normal enough for this test”. At n=6 they have almost no power; at n=600 they flag deviations too small to matter. And choosing the test from a preliminary test on the same data changes the sampling distribution of the test you finally report, so the p-value you quote is no longer the p-value you computed. Rochon and colleagues measured how far off that leaves you; it is not a small correction.

Use what you know about the measurement, look at the data, and default to Welch. If the mean is not the quantity you care about, use a rank test — that is a statement about the question, not about the distribution.

Multiplicity. Six comparisons, one of them at p = 0.04, is not one result; it is six chances to be wrong.

import numpy as np
from scipy import stats

ps = np.array([0.001, 0.008, 0.02, 0.04, 0.2, 0.6])
adjusted = stats.false_discovery_control(ps)        # Benjamini–Hochberg, scipy >= 1.11
print([f"{p:.3f}" for p in adjusted])

['0.006', '0.024', '0.040', '0.060', '0.240', '0.600']

The 0.04 becomes 0.06 and stops being a finding. Benjamini–Hochberg controls the expected proportion of false discoveries among the ones you call, which is the right target for a screen. It does not control the chance of any false discovery at all — that is Bonferroni’s job, and Holm’s is the better version of it.

What I report now

  • n per group, and what one observation is: a patient, a mouse, a well, a fold.
  • The estimate, not just the test: mean difference with a 95% confidence interval, or median and interquartile range for a rank test.
  • The test by name, with the assumption it makes: “Welch’s t-test, which does not assume equal variances”.
  • The p-value to two significant digits, not “p < 0.05” unless it really is.
  • The correction, if there was more than one test.
  • The software and version.

None of this is new statistics. It is the checklist I should have run before anyone asked.

Sources and further reading

Originally written in my notes, October 2020; rewritten and expanded November 2026. This is the first of the notes I said I would move over — the rest are coming.

Written on November 14, 2026