Research: make the belief killable
Introduced| English | Meaning |
|---|---|
| response rate | the share of a sample who actually answer a survey |
| sampling error | the difference between a sample statistic and the population value from chance |
A belief worth £3,600 is worth testing
- The campaign is about to spend real money on the belief that "people want weekly meal bundles". Research exists to make that belief killable — cheaply, before the money spends it.
- The month's spine: 5,000 emailed → 408 responded → 62% said they would try. Three numbers, three different kinds of truth.
The question that can fail
- A research question must be answerable by data you could actually collect: "Do customers love our meals?" cannot fail — it is not research.
- Falsifiable version: "What share of past customers would order the 5-meal bundle at £29 within 30 days?" The 62% intent is now a hypothesis the launch itself will test.
Which kind of error is this?
Sort research defects into sampling error, design bias, and question faults - each with a different repair.
"Do customers love our meals?" is a valid research question because it is simple.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
No data could refute it - a question you cannot be wrong about is not research. Falsifiable, measurable, decidable: rewrite until it is.
Match each study type to what it can honestly deliver.
Difficulty 2/5 · 140 coins + a small bonus on first completion today
Each design answers one question type; asking it the wrong question wastes the study.
The interval and its two hedges
- 62% of 408 → the 95% interval is 62% ± 4.7 points: [57.3%, 66.7%] ✓ — that is the sampling error, and it shrank with n.
- Hedge two: the response rate 响应率 was 8.2% — regulars answer food surveys, so 62% likely overstates the base. Design error does not shrink.
p = 0.62, n = 408. The half-width of the 95% interval, in percentage points (one decimal)?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$1.96\sqrt{0.62 \times 0.38/408} = 0.047$ -> the interval is [57.3%, 66.7%].
Of 5,000 emailed, 408 responded. The response rate, in % (one decimal)?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
$408/5{,}000 = 8.16\%$ - and the design warning rides with it: regulars answered.
Roughly what n would halve the interval's half-width from 4.7 points?
Difficulty 2/5 · 140 coins + a small bonus on first completion today
Halving needs 4x the sample: about 1,632. Precision is bought at root n - the same exchange rate as unit 5.4.
Which repair actually reduces non-response bias?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
More of the same design scales the bias. A random-sample experiment brings the non-respondents' behaviour back into the data.
The 62% and the 13.5% measure the same thing and one of them must be wrong.
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Different questions: attitude versus behaviour, survey versus offer. Both are honest; the campaign is steered by the behavioural one.
Surveys aim; experiments decide
- The A/B repair: offer the bundle to 200 random past customers and watch behaviour — 27 accept (13.5%).
- Intent said 62%; behaviour said 13.5%. The gap is the most replicated finding in market research: stated intention overstates purchase, always.
Intent 62%; the A/B offer to 200 converts 27 (13.5%). The honest reading?
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The intent-to-behaviour gap is the most replicated finding in the field: use the survey to aim, the experiment to decide.
Three research slips
- Quoting 62% bare. Interval and response rate ride along, or the number is a mood.
- Treating responders as the base. Non-respondents differ — in exactly the direction that flatters you.
- Confusing descriptive with causal. A survey describes; only a controlled test explains.
Recap
- Questions must be falsifiable; hypotheses must be testable by the launch.
- Sampling error lives in the interval; design error lives in the response rate.
- Surveys for direction; experiments for truth.