Regression

6.2.1. Regression#

Regression is the supervised learning task of predicting a continuous numerical output from input features. Unlike classification, which assigns labels from a fixed set of categories, regression models produce real-valued predictions - house prices, temperatures, customer lifetime values, demand forecasts, and more.

In regression, we learn a function \(f(X) \rightarrow y\) where \(y \in \mathbb{R}\). The goal is for \(f\) to generalise well: predictions on new, unseen data should be close to the true values.

Common examples:

  • Predicting house prices from square footage, location, and number of rooms

  • Forecasting tomorrow’s temperature from historical weather data

  • Estimating a patient’s blood pressure from lifestyle and clinical features

This chapter covers the following topics:

  1. Metrics & Loss Functions - how we quantify and optimise the error between predictions and ground truth

  2. Linear Regression - the foundational model and interpretable baseline

  3. Regularized Regression - Ridge, Lasso, and Elastic Net for high-dimensional settings

  4. Support Vector Regression - kernel-based non-linear regression

  5. Decision Tree Regression - recursive partitioning and the bias–variance trade-off

  6. Ensemble Methods - Voting, Bagging, and Stacking

  7. Random Forest - optimised bagging with feature randomisation

  8. Boosting - AdaBoost and Gradient Boosting