Neural Networks Basics

The Biological Inspiration#

The artificial neural network borrows its metaphor from the brain. A biological neuron receives signals from thousands of other neurons, integrates them, and fires if the combined signal exceeds a threshold.

The artificial neuron does the same thing with algebra.

The Artificial Neuron (Perceptron)#

A single neuron:

1. Takes inputs $x_1, x_2, \ldots, x_n$
2. Multiplies each by a weight $w_i$ (its importance)
3. Adds a bias $b$ (a constant offset)
4. Passes the result through an activation function

$\text{output} = f\left(\sum_{i=1}^{n} w_i x_i + b\right) = f(\mathbf{w} \cdot \mathbf{x} + b)$

import numpy as np

def neuron(inputs: np.ndarray, weights: np.ndarray, bias: float) -> float:
    z = np.dot(weights, inputs) + bias   # linear combination
    return relu(z)                        # apply activation

def relu(z: float) -> float:
    return max(0, z)                      # ReLU: the most common activation

Activation Functions#

Without an activation function, a neural network is just linear algebra — no matter how many layers you stack, it collapses into a single linear transformation. Activations introduce non-linearity, which is what lets networks learn complex patterns.

Activation	Formula	Use case
ReLU	$\max(0, z)$	Default for hidden layers
Sigmoid	$\frac{1}{1+e^{-z}}$	Binary classification output
Softmax	$\frac{e^{z_i}}{\sum_j e^{z_j}}$	Multi-class output
Tanh	$\frac{e^z - e^{-z}}{e^z + e^{-z}}$	RNNs and some older architectures
GELU	Smooth approximation of ReLU	Transformers

Building a Network#

Stack neurons in layers, and you get a neural network. Information flows forward through layers:

In PyTorch, this is:

import torch
import torch.nn as nn

class SimpleNet(nn.Module):
    def __init__(self, input_size: int, hidden_size: int, output_size: int):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(input_size, hidden_size),   # Input → Hidden
            nn.ReLU(),                             # Activation
            nn.Linear(hidden_size, hidden_size),  # Hidden → Hidden
            nn.ReLU(),
            nn.Linear(hidden_size, output_size),  # Hidden → Output
        )
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.layers(x)

model = SimpleNet(input_size=3, hidden_size=64, output_size=1)
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
# Parameters: 4,353

Backpropagation#

How do weights get updated? Through backpropagation — applying the chain rule of calculus to propagate the error signal backward through the network.

At a high level:

$\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}$

Modern frameworks (PyTorch, TensorFlow/JAX) compute this automatically. You write the forward pass; they derive the backward pass via automatic differentiation.

You almost never need to implement backpropagation by hand. Understandingthat it exists and what it computes is enough for practical work. Understanding how it works gives you intuition for debugging and architecture design.

The Vanishing Gradient Problem#

In deep networks, gradients can shrink exponentially as they propagate backward through many layers. By the time the gradient reaches early layers, it's nearly zero — and those layers stop learning.

Solutions:

• ReLU activations — don't saturate for positive inputs
• Batch Normalisation — normalises activations between layers
• Residual connections — create gradient "highways" that skip layers (key to deep ResNets and Transformers)

class ResidualBlock(nn.Module):
    def __init__(self, size: int):
        super().__init__()
        self.block = nn.Sequential(
            nn.Linear(size, size),
            nn.LayerNorm(size),
            nn.ReLU(),
            nn.Linear(size, size),
            nn.LayerNorm(size),
        )
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return x + self.block(x)  # ← residual connection

How Deep is "Deep"?#

"Deep" learning simply means more than one hidden layer. In practice:

Network type	Typical depth	Application
Shallow MLP	1–2 hidden layers	Tabular data
CNN	10–50 layers	Image recognition
ResNet	50–152 layers	Image classification
Transformer (GPT-4)	96+ layers	Language modelling

More depth doesn't always mean better. Start shallow, add complexity only when you have evidence the model is underfitting.

Summary#

Neural networks are function approximators built from stacked layers of neurons. They learn by adjusting weights via backpropagation to minimise a loss function. The depth and non-linearity are what enable them to capture complex patterns that simpler models cannot. In the final lesson, we'll put this knowledge into practice and train a real model.