Testing: surprise, measured
Introduced| English | Meaning |
|---|---|
| null hypothesis/nʌl haɪˈpɒθəsɪs/ | The default claim about a population parameter that a statistical test assesses against an alternative. |
| p-value/piː ˈvæljuː/ | the probability of data at least this extreme if the null hypothesis were true |
Would the data surprise you if nothing were going on?
- "Is the mean really 30 minutes?" A test asks whether the sample would be extreme under a boring default — and refuses to say more than that.
- The discipline: state the default, choose your error budget, compute, and report the effect size. Anything less is ritual.
Set up the machinery
- The null hypothesis 零假设 $H_0$ (mean = 30) against the alternative (mean ≠ 30). Statistic: $t = (\bar x - \mu_0)/SE$.
- The p-value p值: chance of data at least this extreme IF $H_0$ were true. Small p = the data embarrass the default — nothing more.
The course's own test
- Seed-11 sample: $\bar x = 28.66$, $SE = 1.175$ → $t = (28.66 - 30)/1.175 = \mathbf{-1.14}$.
- Threshold at α = 0.05 (two-sided, 15 df): $|t| > 2.131$. Since 1.14 < 2.131: fail to reject — consistent with 30, and with plenty else ✓.
Sample mean 28.66, mu0 = 30, SE 1.175. The t statistic, to two decimals?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$(28.66 - 30)/1.175 = -1.14$ - inside the +/-2.131 acceptance region at alpha 0.05, 15 df.
Two-sided test, alpha 0.05, 15 df. What must |t| exceed to reject?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
The t(15) critical value is 2.131. Our 1.14 does not reach it: fail to reject.
Two errors, one budget
- Type I: rejecting a true null (rate α — YOUR choice). Type II: keeping a false null (rate β); power = 1 − β.
- If free desserts ride on the call, α = 0.10 may be right; if re-training does, 0.01. α is a business decision wearing a Greek letter.
What does this test result license?
Sort test outcomes into what they do and do not allow you to conclude.
A rider's 16 runs: mean 26.9, s = 4.0, shop mean 28.7. The t statistic, to two decimals?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$t = (26.9 - 28.7)/(4.0/4) = -1.8/1.0 = -1.8$ - p about 0.09 two-sided: not significant at 0.05, reportable as suggestive.
p = 0.09 against alpha = 0.05. The correct summary?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Not significant is a statement about THESE data under the null. The 3-minute effect may still be real - the evidence is just short.
When is choosing alpha = 0.10 (instead of 0.05) the RIGHT call?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
alpha is the Type I budget: raise it when false alarms are cheap relative to missed effects. It is a cost decision, not a convention.
p = 0.03 means there is a 3% chance the null hypothesis is true.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
p is the chance of DATA this extreme IF the null is true - a conditional about the data, not a probability about hypotheses.
Significance is not importance
- With n = 400, a 0.4-minute difference is p < 0.01 — statistically significant, operationally invisible (24 seconds!).
- With n = 8, a 3-minute difference can sit at p = 0.3 — invisible to the test, obvious to every customer. Always report the effect size with its interval: "0.4 min, 95% CI [0.1, 0.7] — significant and trivial".
n = 400 shows 0.4 minutes at p = 0.008. The best reading?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Big samples find tiny differences. Significance answers existence; the effect size with its CI answers relevance - 0.4 minutes is 24 seconds.
Failing to reject at alpha = 0.05 proves the null hypothesis true.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The data were not surprising under the null - that is all. A bigger sample might reject; absence of evidence is not evidence of absence.
Three testing slips
- "Accepting" the null. You failed to reject; absence of evidence is not evidence of absence.
- Reading p as the probability the hypothesis is true. p is about the data under the null, nothing else.
- p-hacking. Testing until something clears 0.05 — the subject of the final lesson.
Recap
- $t = (\bar x - \mu_0)/SE$; reject past the threshold chosen as your error budget.
- Fail-to-reject is a result. p describes the data, not the hypothesis.
- Significance asks "is it there?"; the effect size asks "does it matter?"