How much maths do you really need?

Use NumPy: def cosine_similarity(a, b): return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)). This computes the dot product of the vectors divided by the product of their magnitudes. The result ranges from -1 (opposite) to 1 (identical direction). Add epsilon to the denominator to prevent division by zero.

A scalar is a single number (0D), a vector is a 1D array of numbers, a matrix is a 2D array, and a tensor is a generalisation to any number of dimensions. Each is a special case of a tensor. In ML, model weights are stored as tensors, and frameworks like PyTorch and NumPy operate on these n-dimensional arrays.

The dot product of two vectors is the sum of their element-wise products, producing a scalar. It measures similarity between vectors. In neural networks, each neuron computes a dot product between its weights and inputs, followed by a bias and activation function. Matrix multiplication, which powers layer computations, is built from dot products.

Gradient descent is an iterative optimisation algorithm that updates parameters in the direction opposite to the gradient of the loss function. The gradient points in the direction of steepest ascent, so moving against it reduces the loss. The learning rate controls step size. Variants include SGD, Adam, and AdaGrad, each handling learning rates differently.

Softmax converts a vector of real numbers (logits) into a probability distribution by exponentiating each element and normalising by the sum: softmax(z_i) = exp(z_i) / sum(exp(z_j)). It ensures outputs are positive and sum to 1, making them interpretable as class probabilities. It is used in the final layer of multi-class classification networks.

A probability distribution describes how probabilities are assigned to different outcomes. The Gaussian (normal) distribution is bell-shaped, defined by mean and variance, and arises naturally due to the Central Limit Theorem. The Bernoulli distribution models binary outcomes with probability p for success. These are foundational in ML for modelling data and defining loss functions.

Correlation means two variables tend to change together but does not imply one causes the other. Causation requires a direct mechanism. ML models primarily learn correlations, which can lead to spurious predictions when the underlying causal relationship changes. Understanding this distinction is critical for model robustness, fairness, and deployment in high-stakes domains.

Cosine similarity measures the angle between vectors (direction), ranging from -1 to 1. Euclidean distance measures the straight-line distance between points (magnitude matters). Use cosine similarity when only direction matters (text embeddings, recommendation systems). Use Euclidean distance when magnitude is meaningful (spatial data, anomaly detection based on deviation).

A deterministic model always produces the same output for the same input. A stochastic model incorporates randomness and may produce different outputs each time. Most ML training is stochastic (random initialisation, mini-batch sampling, dropout). Stochasticity often helps by preventing overfitting and escaping local minima during optimisation.

Parametric models have a fixed number of parameters regardless of data size (e.g., linear regression, logistic regression). Non-parametric models grow in complexity with more data (e.g., k-NN, decision trees). Parametric models are faster but may underfit; non-parametric models are more flexible but can overfit.

Overfitting occurs when a model memorises training data and performs poorly on unseen data, detected by a large gap between training and validation accuracy. Underfitting means the model is too simple to capture patterns, shown by poor performance on both sets. Regularisation, dropout, and cross-validation address overfitting; increasing model complexity or features addresses underfitting.

The bias-variance tradeoff describes the tension between a model's ability to fit training data (low bias) and its ability to generalise to new data (low variance). High bias leads to underfitting while high variance leads to overfitting. The goal is to find the optimal complexity that minimises total error.

An activation function determines whether a neuron should fire by applying a transformation to the weighted sum of inputs. Without non-linear activations, a multi-layer network would collapse to a single linear transformation, unable to learn complex patterns. Common non-linear activations include ReLU, sigmoid, and tanh.

Backpropagation is an algorithm that computes the gradient of the loss function with respect to each weight by applying the chain rule layer by layer from output to input. These gradients tell the optimiser how to adjust weights to minimise loss. It makes training deep networks computationally feasible.

Batch gradient descent computes gradients over the entire dataset per update, which is stable but slow. Stochastic gradient descent (SGD) updates weights after each sample, which is noisy but fast. Mini-batch gradient descent is the practical middle ground, computing gradients over small batches (32-512 samples), offering both speed and stability.

Regularisation adds a penalty term to the loss function to prevent overfitting. L1 (Lasso) adds the absolute value of weights, encouraging sparsity and effective feature selection. L2 (Ridge) adds the squared weights, shrinking all weights toward zero without eliminating them. Elastic Net combines both approaches.

Attention computes a weighted sum of Value vectors, where the weights are determined by the compatibility between Query and Key vectors. For each token, Q represents what it is looking for, K represents what it offers, and V represents the actual content. The attention score is computed as softmax(QK^T / sqrt(d_k)) * V.

Eigenvectors of a matrix are vectors whose direction remains unchanged when multiplied by that matrix; eigenvalues are the scaling factors. In ML, they are central to PCA for dimensionality reduction, spectral clustering, understanding covariance structures, and analysing the stability of dynamical systems and recurrent networks.

The chain rule states that the derivative of a composite function f(g(x)) is f'(g(x)) * g'(x). In backpropagation, each layer in a neural network is a composed function, and the chain rule enables computing how the loss changes with respect to each parameter by multiplying local gradients along the computation path.

Bayes' theorem states P(A|B) = P(B|A) * P(A) / P(B), relating posterior probability to likelihood, prior, and evidence. In ML, Naive Bayes classifiers use it for text classification. Bayesian inference updates beliefs about model parameters as new data arrives. It is also fundamental to probabilistic graphical models and Bayesian neural networks.

PCA finds the directions (principal components) of maximum variance in the data by computing the eigenvectors of the covariance matrix. Data is projected onto the top-k eigenvectors to reduce dimensionality while preserving the most variance. Mathematically, it is equivalent to SVD of the centered data matrix. It assumes linear relationships between features.

The L1 norm is the sum of absolute values of vector elements, while the L2 norm is the square root of the sum of squared elements. L1 regularisation (Lasso) promotes sparsity by driving weights to exactly zero, useful for feature selection. L2 regularisation (Ridge) distributes penalty evenly, shrinking all weights. L1 has diamond-shaped constraint regions; L2 has circular ones.

Cross-entropy loss measures the difference between predicted probability distribution and true distribution: -sum(y_i * log(p_i)). It is preferred over MSE for classification because it penalises confident wrong predictions heavily, has better gradient properties (avoids saturation with sigmoid/softmax), and is derived from maximum likelihood estimation for categorical distributions.

In CNNs, convolution slides a learnable kernel (filter) across the input, computing element-wise multiplication and summing at each position. Technically it is cross-correlation (no kernel flip). For a 2D input, output[i,j] = sum(input[i+m, j+n] * kernel[m,n]). This operation extracts local features like edges, textures, and patterns while sharing parameters across spatial positions.

Adam maintains two exponential moving averages: first moment (mean of gradients, like momentum) and second moment (mean of squared gradients, like RMSProp). It adapts per-parameter learning rates using these estimates with bias correction. The update rule is: theta -= lr * m_hat / (sqrt(v_hat) + epsilon). This combination makes it robust across different architectures.

The CLT states that the mean of many independent random variables tends toward a normal distribution, regardless of the original distribution. In ML, this justifies using Gaussian assumptions in many models, underpins statistical testing of model performance, explains why batch normalisation works, and is why many natural phenomena are approximately normally distributed.

A loss function quantifies the discrepancy between model predictions and true values. For regression, use MSE (sensitive to outliers) or MAE (robust). For classification, use cross-entropy. For ranking, use hinge loss or triplet loss. For imbalanced data, use focal loss or weighted cross-entropy. The choice depends on the task, data distribution, and what errors are most costly.

As dimensions increase, data becomes increasingly sparse, and distance metrics become less meaningful because points tend to be equidistant. This makes nearest-neighbour methods ineffective, increases the data needed for reliable estimation exponentially, and complicates optimisation. Dimensionality reduction techniques like PCA, t-SNE, or autoencoders help mitigate this problem.

Batch normalisation normalises each layer's inputs to zero mean and unit variance across the mini-batch, then applies learnable scale and shift parameters. Mathematically, it reduces internal covariate shift, smooths the loss landscape, and allows higher learning rates. It acts as a regulariser due to the noise introduced by per-batch statistics.

Entropy measures the uncertainty or randomness in a probability distribution: H = -sum(p_i * log(p_i)). Maximum entropy means maximum uncertainty (uniform distribution). In decision trees, information gain (reduction in entropy) is used to select the best feature to split on at each node, choosing the feature that most reduces uncertainty about the target variable.

The learning rate determines the step size in gradient descent: theta_new = theta_old - lr * gradient. Too high a learning rate causes oscillation or divergence (overshooting minima). Too low makes training prohibitively slow and can get stuck in poor local minima. The optimal learning rate depends on the loss curvature, which varies throughout training.

LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices into each layer. Instead of updating a full weight matrix W (d x d), it learns two smaller matrices A (d x r) and B (r x d) where r is much smaller than d. This reduces trainable parameters by 100-1000x while maintaining performance. Multiple LoRA adapters can be swapped for different tasks.

The learning rate schedule controls how the learning rate changes during training. For fine-tuning, a warmup phase (5-10% of steps) gradually increases from zero to prevent large initial weight updates. Followed by cosine or linear decay to allow fine-grained convergence. The peak learning rate for fine-tuning is typically 10-100x lower than pre-training to preserve learned representations.

The vanishing gradient problem occurs when gradients become extremely small as they propagate through many layers, effectively stopping learning in earlier layers. It is addressed by using ReLU activation functions instead of sigmoid/tanh, batch normalisation, residual connections (skip connections), and proper weight initialisation techniques like He or Xavier initialisation.

In convex optimisation, any local minimum is the global minimum, guaranteeing convergence. Neural network loss landscapes are non-convex with many local minima and saddle points. This means gradient descent may get stuck. In practice, research shows that most local minima in high-dimensional spaces have similar loss values, and SGD's noise helps escape saddle points.

The Jacobian is a matrix of all first-order partial derivatives of a vector-valued function. In deep learning, it represents how each output dimension changes with respect to each input dimension. It is used in backpropagation through layers, normalising flows, and analysing the sensitivity of model outputs. The Jacobian determinant measures volume change under transformations.

KL divergence measures how one probability distribution Q differs from a reference distribution P: KL(P||Q) = sum(P(x) * log(P(x)/Q(x))). It is always non-negative and zero only when P=Q. Cross-entropy equals the entropy of P plus the KL divergence from P to Q. Minimising cross-entropy is equivalent to minimising KL divergence when P is fixed.

For binary cross-entropy L = -[y*log(sigma(z)) + (1-y)*log(1-sigma(z))], where sigma is sigmoid. Using the fact that sigma'(z) = sigma(z)(1-sigma(z)), the gradient dL/dz simplifies to sigma(z) - y, which is the predicted probability minus the true label. This elegant simplification is why sigmoid + BCE is commonly used.

SVD decomposes any matrix A into three matrices: A = U * Sigma * V^T, where U and V are orthogonal and Sigma is diagonal with singular values. In ML, it is used for PCA, matrix factorisation in recommendation systems, low-rank approximation for model compression, and latent semantic analysis in NLP.

MLE finds parameter values that maximise the probability of observing the training data. For a classification network with softmax output, maximising the likelihood of correct labels is equivalent to minimising cross-entropy loss. For regression with Gaussian noise, MLE is equivalent to minimising MSE. This probabilistic interpretation connects loss functions to statistical theory.

The Hessian is the matrix of all second-order partial derivatives of the loss function. It captures the curvature of the loss landscape and can inform optimal step sizes. For a network with n parameters, the Hessian has n^2 entries, making it infeasible to compute for large models. Approximations like Fisher information matrix, KFAC, or diagonal Hessian are used instead.

The reparametrisation trick transforms sampling from a learned distribution into a deterministic function of the parameters plus noise: z = mu + sigma * epsilon, where epsilon is from N(0,1). This makes the sampling operation differentiable, enabling backpropagation through the stochastic layer. Without it, VAEs cannot be trained end-to-end with gradient descent.