Skills101

AdvancedPython for Data Science & Machine Learning · Lesson 15 of 15 · Updated ·

Hypothesis Testing — t-Tests, Paired Tests, Chi-Square, p-Values and Type I/II Errors

The statistics lesson showed one A/B test. Real questions come in a few standard shapes: Do two groups differ? Did the same students improve? Are two categories related? Each shape has a standard hypothesis test. This lesson computes each one by hand (so you understand it), and names the library function you would use at work.

1. The five steps

  1. Null hypothesis (H₀) — "nothing is going on": no difference, no effect, no relationship.
  2. Alternative hypothesis (H₁) — what you suspect: there is a difference.
  3. Significance level (α) — chosen before looking at results; usually 0.05.
  4. Test statistic and p-value — how surprising the data would be if H₀ were true.
  5. Decision — if p < α, reject H₀; otherwise you fail to reject it. (Not "prove H₀" — absence of evidence is not evidence of absence.)

2. Two groups: the two-sample t-test

Question: a college offered an optional aptitude coaching course. Did students who took it score higher on the mock placement test?

The t-statistic is the difference in means divided by its standard error — "how many standard errors apart are the two groups?" The version below (Welch's t-test) does not assume both groups have the same spread.

import numpy as np

coached = np.array([68, 74, 71, 80, 77, 69, 83, 75, 72, 78, 81, 70])
not_coached = np.array([65, 70, 62, 73, 68, 66, 71, 64, 69, 67, 74, 63])

def welch_t(a, b):
    se = np.sqrt(a.var(ddof=1) / len(a) + b.var(ddof=1) / len(b))
    return (a.mean() - b.mean()) / se

t_obs = welch_t(coached, not_coached)
print(f"means: {coached.mean():.2f} vs {not_coached.mean():.2f}, t = {t_obs:.2f}")

# p-value by permutation: if coaching made no difference, the labels are arbitrary — shuffle them
rng = np.random.default_rng(0)
both = np.concatenate([coached, not_coached])
count = 0
for _ in range(10_000):
    rng.shuffle(both)
    if abs(welch_t(both[:12], both[12:])) >= abs(t_obs):     # two-sided: a difference in either direction
        count += 1
print("p-value:", count / 10_000)
means: 74.83 vs 67.67, t = 3.93
p-value: 0.0011

Histogram of the t-statistic from 10,000 random shufflings of the group labels, centred on zero, with the observed t of about 3.9 far out in the right tail

The shuffled t-values show what "no effect" looks like; our observed t is far out in the tail, so the p-value is tiny and we reject H₀. At work you would call scipy.stats.ttest_ind(coached, not_coached, equal_var=False), which gives the same t and a p-value from the t-distribution instead of shuffling.

But be careful with the conclusion. Students chose to take coaching; motivated students may score higher anyway. The test says the groups differ — not that coaching caused it. Only random assignment (an experiment) supports a causal claim.

3. Same students twice: the paired test

Question: ten students took a mock test, did two weeks of practice, then took another. Did they improve?

These are not two independent groups — each "after" score belongs to a "before" score. Analyse the differences:

import numpy as np

before = np.array([52, 61, 47, 70, 58, 66, 49, 55, 63, 60])
after  = np.array([58, 63, 55, 71, 64, 65, 56, 60, 70, 62])
d = after - before
t = d.mean() / (d.std(ddof=1) / np.sqrt(len(d)))
print("differences:", d)
print(f"mean improvement {d.mean():.1f} marks, paired t = {t:.2f}")

# permutation for paired data: under H0 each difference is equally likely to be + or −
rng = np.random.default_rng(1)
signs = rng.choice([-1, 1], size=(10_000, len(d)))
fake = (signs * d).mean(axis=1)
print("p-value:", np.mean(np.abs(fake) >= abs(d.mean())))
differences: [ 6  2  8  1  6 -1  7  5  7  2]
mean improvement 4.3 marks, paired t = 4.45
p-value: 0.0055

Pairing removes the big differences between students (some always score 50, others 70) and keeps only the change within each student. That makes the test far more sensitive. The library function is scipy.stats.ttest_rel(after, before). Treating paired data as two independent groups is a classic mistake.

4. Two categories: the chi-square test

Question: does branch relate to whether students prefer online or offline classes? The data is a table of counts, so means don't apply. The chi-square (χ²) test of independence compares the observed counts with the counts we would expect if the two variables were unrelated:

expected = row total × column total ÷ grand total, and χ² = Σ (observed − expected)² ÷ expected

import numpy as np

observed = np.array([[45, 35],     # CSE: online, offline
                     [25, 35],     # ECE
                     [15, 45]])    # ME
row, col, total = observed.sum(axis=1, keepdims=True), observed.sum(axis=0, keepdims=True), observed.sum()
expected = row * col / total
chi2 = ((observed - expected) ** 2 / expected).sum()
print("expected counts if unrelated:\n", expected.round(1))
print(f"chi-square = {chi2:.2f}")

# p-value by permutation: rebuild one row per student, shuffle the preference column
branch = np.repeat([0, 0, 1, 1, 2, 2], observed.ravel())
pref = np.tile([0, 1], 3).repeat(observed.ravel())
rng = np.random.default_rng(2)
bigger = 0
for _ in range(5000):
    rng.shuffle(pref)
    o = np.zeros((3, 2))
    np.add.at(o, (branch, pref), 1)
    bigger += ((o - expected) ** 2 / expected).sum() >= chi2
print("p-value:", bigger / 5000)
expected counts if unrelated:
 [[34.  46. ]
 [25.5 34.5]
 [25.5 34.5]]
chi-square = 13.73
p-value: 0.0014

ME students prefer offline far more than the "unrelated" model expects, and the p-value is well below 0.05, so branch and preference are related. The library function is scipy.stats.chi2_contingency(observed). Rule of thumb: the formula-based version needs expected counts of about 5 or more in every cell.

Data shapeTestSciPy function
two independent groups, numericWelch's t-testttest_ind(a, b, equal_var=False)
same subjects twicepaired t-testttest_rel(after, before)
more than two groups, numericone-way ANOVAf_oneway(a, b, c)
two categorical variableschi-square testchi2_contingency(table)
skewed data, small samplesMann–Whitney Umannwhitneyu(a, b)

5. Two ways to be wrong

A two-by-two table: rejecting a true null hypothesis is a Type I error, a false alarm with probability alpha; failing to reject a false null is a Type II error, a missed effect with probability beta; power is one minus beta

  • Type I error (false positive) — you "find" an effect that isn't there. Its probability is α (5% if α = 0.05).
  • Type II error (false negative) — you miss a real effect. Its probability is β.
  • Power = 1 − β — the chance of detecting a real effect. It grows with sample size and with the size of the effect.

Lowering α to 0.01 cuts false alarms but increases misses, unless you collect more data.

6. The multiple-testing trap

Test 20 things that have no real effect at α = 0.05 and you still expect about one "significant" result:

alpha, tests = 0.05, 20
print(f"chance of at least one false positive: {1 - (1 - alpha) ** tests:.0%}")
print(f"Bonferroni-corrected alpha per test: {alpha / tests}")
chance of at least one false positive: 64%
Bonferroni-corrected alpha per test: 0.0025

If you check many metrics, segments or features, either correct for it (the simple Bonferroni correction divides α by the number of tests) or decide the one main metric before the experiment.

Finally, significant is not the same as important. With a million users, a 0.01-mark difference can have p < 0.001 and still not matter to anyone. Always report the size of the effect (with a confidence interval) next to the p-value.

Common mistakes

  • Saying "p > 0.05, so there is no difference" — you only failed to find one.
  • Using an independent-groups test on before/after data from the same people.
  • Claiming causation from groups that chose themselves.
  • Testing many things and reporting only the one with p < 0.05.
  • Reporting p-values without the effect size.

Quick recap

  • Steps: H₀, H₁, α (before looking), statistic and p-value, decision.
  • Two groups → Welch's t-test; same people twice → paired test; counts in a table → chi-square.
  • A permutation test builds the "no effect" distribution by shuffling labels or signs.
  • Type I = false alarm (α); Type II = missed effect (β); power = 1 − β.
  • Correct for multiple tests, and report effect size, not just p.

Practice what you learned

📝 Quiz: Hypothesis Testing5 questions · instant answers & explanations · +10 XP · counts towards your free certificateTake the quiz →