Model Validation and Business Communication

You have one or more candidate models and must produce reliable estimates of test error.

Split the data into training and test sets (typically 70/30 or 80/20). Fit and tune on the training set using k-fold CV; report the final held-out error on the test set. For small datasets, lean on k-fold CV directly. Never tune on the test set — that contaminates the error estimate.

Tune on train (via CV); report on test (untouched).

A model produces predictions on a holdout set and you must compute and interpret the relevant performance metric.

Regression: RMSE for L2-error sensitivity; MAE for outlier-robust error. Classification: AUC for ranking quality across thresholds; log loss for probability calibration; accuracy only when classes are balanced. Use confusion matrix, precision, recall, and F1 when discussing thresholds.

Match the metric to the business objective; never report accuracy alone for imbalanced classes.

You have completed your modeling work and must write the executive summary for a business reader.

Open with the business problem and your recommendation. State the data, the final model, and key validation metrics in plain English. Discuss assumptions, limitations, and next steps. Avoid jargon when possible; when you must use it, define it briefly. Make sure every chart has a one-line caption stating the takeaway.

Executive summary structure: problem → recommendation → method → validation → limitations.