Supervised Learning

What Makes it "Supervised"?#

Supervised learning gets its name from the idea of a teacher providing correct answers during training. The model sees an input, makes a prediction, compares it to the correct answer, and adjusts.

This feedback loop is the engine of all supervised ML.

The Training Loop#

Every supervised learning algorithm follows the same fundamental cycle:

1. Forward pass   → model makes a prediction
2. Compute loss   → measure how wrong the prediction is
3. Backward pass  → compute gradients (which direction to move?)
4. Update weights → take a small step in the right direction
5. Repeat

In code, with PyTorch:

import torch
import torch.nn as nn

model = nn.Linear(3, 1)          # 3 features → 1 output
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.MSELoss()

for epoch in range(1000):
    # Forward pass
    predictions = model(X_train)
    
    # Compute loss
    loss = loss_fn(predictions, y_train)
    
    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    
    # Update weights
    optimizer.step()

print(f"Final loss: {loss.item():.4f}")

Regression vs Classification#

These are the two main tasks in supervised learning.

Regression: Predicting Continuous Values#

The output is a number on a continuous scale.

Examples:

• House price given square footage, location, year built
• Temperature tomorrow given historical weather
• Sales revenue next quarter

Loss function: Mean Squared Error (MSE)

$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

Where $y_i$ is the true value and $\hat{y}_i$ is the predicted value.

Classification: Predicting Categories#

The output is a discrete class label.

Examples:

• Email: spam or not spam
• Image: cat, dog, or bird
• Tumour: malignant or benign

Loss function: Cross-Entropy Loss

$\mathcal{L} = -\sum_{c=1}^{C} y_c \log(\hat{p}_c)$

Where $C$ is the number of classes, $y_c$ is 1 if class $c$ is correct, and $\hat{p}_c$ is the predicted probability.

Binary classification (two classes) usesBinary Cross-Entropy. Multi-class classification uses Categorical Cross-Entropy. Most libraries handle this distinction automatically if you use the right output layer.

Gradient Descent#

How does the model adjust its weights? Through gradient descent — following the slope of the loss surface downhill.

weights = weights - learning_rate × gradient_of_loss

$\theta_{t+1} = \theta_t - \alpha \nabla_\theta \mathcal{L}(\theta_t)$

The learning rate $\alpha$ controls how big each step is:

Too small	Too large
Slow convergence	Divergence (loss explodes)
Gets stuck in local minima	Oscillates, never settles

In practice, Adam (Adaptive Moment Estimation) is the default optimiser. It adapts the learning rate for each parameter and usually works well without tuning.

Bias–Variance Trade-off#

This is one of the most important concepts in all of machine learning.

• High bias (underfitting): Model is too simple to capture patterns. Bad on training and test data.
• High variance (overfitting): Model memorises training data. Great on training, terrible on test data.
• Goal: Find the sweet spot — a model complex enough to capture real patterns, simple enough to generalise.

Regularisation#

Regularisation adds a penalty to the loss function for large weights, discouraging overfitting:

Technique	Idea	When to use
L2 (Ridge)	Penalise sum of squared weights	Most regression tasks
L1 (Lasso)	Penalise sum of absolute weights	Feature selection
Dropout	Randomly zero out neurons during training	Neural networks
Early stopping	Stop training when validation loss stops improving	All deep learning

Evaluating Your Model#

Never evaluate on training data alone. Always use a held-out test set.

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, mean_squared_error

# Split: 80% train, 20% test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model.fit(X_train, y_train)

# Evaluate on unseen data
predictions = model.predict(X_test)
accuracy = accuracy_score(y_test, predictions)
print(f"Test accuracy: {accuracy:.2%}")

Usecross-validation instead of a single train/test split when your dataset is small. It gives a more reliable estimate by training and evaluating on multiple splits of the data.

Summary#

Supervised learning works by training a model on labelled examples, minimising a loss function via gradient descent. The key is generalisation — building a model that performs well on new data, not just the training set. In the next lesson, we'll see how neural networks dramatically expand what's possible.