Basics of Statistical Learning

A modeling scenario is described and you must identify whether it is supervised regression, supervised classification, unsupervised clustering, or dimension reduction.

Ask first whether there is a labeled response variable. If yes, it is supervised — then ask whether the response is continuous (regression) or categorical (classification). If no labels, it is unsupervised — then determine whether the goal is grouping (clustering) or compression (dimension reduction). Match the modeling method to the task type.

Supervised → labels provided; Unsupervised → no labels.

You are given the relationship between test error and model complexity and must explain why test error has a U-shape.

Decompose test MSE: E[(y - ŷ)^2] = Bias(ŷ)^2 + Var(ŷ) + σ^2 (irreducible). As model flexibility increases, bias falls but variance rises. The optimal complexity minimizes the sum. Recognize that training error always decreases with complexity, but test error U-shapes — the gap is overfitting.

E[(y - ŷ)^2] = Bias^2 + Variance + σ^2 (irreducible).

You must estimate test error or standard error of a parameter using a resampling method.

For k-fold CV: split data into k folds, train on k-1 and test on 1, repeat for each fold and average. Higher k → lower bias, higher variance. Bootstrap: sample with replacement to build B bootstrap samples and compute the statistic on each; the empirical distribution of bootstrap statistics estimates standard error. Use LOOCV (k = n) for small data sets where bias matters more than variance.

k-fold CV: average test error across k folds; Bootstrap: B resamples with replacement.