Linear Models

A regression output with coefficients, standard errors, R^2, and residual plots is given, and you must interpret coefficients, evaluate fit, and check assumptions.

Read each coefficient as the change in expected response per unit change in the predictor, holding other predictors fixed. Check t-statistics for significance and the F-statistic for overall fit. Evaluate residual plots for linearity, homoscedasticity, normality, and independence. Beware multicollinearity when correlated predictors yield large standard errors.

β̂ = (X^T X)^{-1} X^T y; R^2 = 1 - RSS / TSS.

A response variable does not look normal (count, binary, positive-skewed) and you must choose a GLM and interpret its coefficients.

Pick the distribution to match the response (Poisson for counts, binomial for binary, gamma for positive skewed) and the link function for interpretability. Log link gives multiplicative effects; logit gives log-odds. Fit by IRLS and interpret coefficients on the link scale (or exponentiate for multiplicative effects). Check fit with deviance and Pearson residuals.

g(E[Y]) = X β; common links: identity (normal), log (Poisson), logit (binomial).

A model has many candidate predictors and you must apply a variable selection procedure to choose a parsimonious subset.

Forward: start with no predictors, add the one that most improves a criterion (AIC, BIC) at each step. Backward: start with all predictors, remove the one whose removal most improves the criterion. Stepwise: combine adding and removing at each step. Penalty terms differ: BIC penalizes complexity more heavily than AIC, yielding sparser models.

AIC = 2k - 2 ln L; BIC = k ln(n) - 2 ln L. Lower is better.

A high-dimensional regression is unstable due to multicollinearity, and you must apply a regularization method.

Ridge: add λ Σ β_j^2 to the loss — shrinks coefficients toward zero but rarely to exactly zero. Lasso: add λ Σ |β_j| — produces sparse solutions where some coefficients are exactly zero. Elastic net: combines both penalties for grouping correlated predictors while inducing sparsity. Tune λ via cross-validation.

Ridge: L2 penalty (shrinkage). Lasso: L1 penalty (sparsity). Elastic net: convex combination.