Regression and classification
Introduced
Vocabulary
| English | Meaning |
|---|---|
| linear regression/ˈlɪnɪə rɪˈɡreʃn/ | A method of modelling a response using a linear combination of explanatory variables and fitted coefficients. |
| logistic regression | A model mapping a linear score through a logistic function to estimate a binary-class probability. |
| decision boundary | The boundary between input regions assigned different predicted classes. |
Lines and boundaries
- Linear regression fits $\hat{y} = wx + b$ by least squares; logistic regression fits a probability boundary with the sigmoid; the decision boundary is where the model changes its answer.
{exercise:uni-cs1304-models}
Practice
The decision boundary is where —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The score crosses zero: the answer flips.
Least squares, closed form
- Course points $(1,2), (2,3), (3,5), (4,4), (5,6)$: $\bar{x} = 3$, $\bar{y} = 4$, $w = 9/10 = 0.9$, $b = 1.3$ — residuals $-0.2, -0.1, 1.0, -0.9, 0.2$, SSE $1.9$ (every number suite-verified).
- Logistic boundary with $w_1 = 2, w_2 = -1, b = -1$: the line $x_2 = 2x_1 - 1$; a parcel at $(2, 2)$ scores $+1$ (late), at $(2, 5)$ scores $-2$ (on time); the margin is the score's magnitude.
{exercise:uni-cs1304-leastsquares}
Practice
The course five points fit the line —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
w = 9/10, b = 4 - 0.9 x 3 = 1.3; SSE 1.9.
The overfit, measured
- A 4th-degree polynomial hits SSE $0$ on the five training points but SSE $\approx 42.7$ on the five held-out points — against the line's $0.55$. Training error is an advertisement; held-out error is the product.
- Model choice by held-out error; and fitted weights are not causes (unit 8's confounding lives here too).
{exercise:uni-cs1304-overfit}
Practice
The 4th-degree polynomial on held-out data scores SSE —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Training 0, held-out 42.7 vs the line 0.55 - the overfit bill.