Reading sources critically
| English | Chinese | Pinyin |
|---|---|---|
| operationalisation | 操作化 | cāo zuò huà |
| authority | 权威性 | quán wēi xìng |
| currency | 时效性 | shí xiào xìng |
| bias | 偏见 | piān jiàn |
| missing data | 缺失数据 | quē shī shù jù |
A statistic is a measurement of a definition
- Change the definition of "unemployed" and the unemployment rate changes without anything happening in the world.
- Operationalisation 操作化 is turning an abstract idea into something measurable, and the choice is always arguable.
- So the first question about any number is not "is it accurate" but "what exactly was counted".
Two sources give youth unemployment as 8% and 21%. What is the most likely reason?
Neither is a lie. Quoting one without its definition is choosing a fact, not reporting one.
Judging a source
- Authority 权威性: who produced it, and what do they know?
- Currency 时效性: when, and does that matter for this claim?
- Bias 偏见: what does the author want, and who paid for the work?
- In social science the bias is often in what was measured rather than in what is said, which is harder to spot and more consequential.
Match each test to the question it asks.
In social science the bias is often in what was measured, which is harder to see than what is said.
What was counted
- Ask how the categories were defined, who was included, and who could not be reached.
- A survey of internet users describes internet users. A helpline's records describe people who phoned.
- Missing data 缺失数据 is rarely missing at random, and the people absent from a dataset are usually absent for a reason connected to the topic.
People missing from a dataset are usually missing at random.
They are usually absent for a reason connected to the topic, which is exactly what biases the result.
Write one question you would ask about a survey reporting that "70% of students feel stressed".
Example: "How was 'stressed' defined in the question, and what were students actually asked?"
Two true statements about the same country.
Youth unemployment is 8%. Counting people actively seeking work in the last four weeks.
Youth unemployment is 21%. Counting everyone not in work, education or training.
Neither is a lie. They operationalise "unemployed" differently, and an argument that quotes one without the definition is not reporting a fact — it is choosing one.
Ask what was counted before you ask whether it is high. Almost every disagreement about social statistics is a disagreement about definitions wearing the costume of a disagreement about facts.
Before comparing two countries' crime figures, what must you check?
Definitions and recording practices differ, and a comparison built on that is worse than none.
Do not compare two statistics from different sources without checking they counted the same thing. Two countries' "literacy rates" or "crime figures" are frequently not comparable at all, and a comparison built on them is worse than no comparison.