Machine learning data

normalizing training sets

When inputting data to a deep learning model, it is standard practice to normalize the data to zero mean and unit variance. What does this mean and why do we do this?

Let’s say the input data consists of several features x1, x2,…xn. Each feature might have a different range of values. For instance, values for feature x1 might range from 1 through 5, while values for feature x2 might range from 1000 to 99999.

So, for each feature column separately, we take the values of all samples in the dataset and compute the mean and the variance. And then normalize the values using the formula below.

To understand what happens without normalization, let’s look at an example with just two features that are on drastically different scales. Since the network output is a linear combination of each feature vector, this means that the network learns weights for each feature that are also on different scales. Otherwise, the large feature will simply drown out the small feature.

Then during gradient descent, in order to “move the needle” for the Loss, the network would have to make a large update to one weight compared to the other weight. This can cause the gradient descent trajectory to oscillate back and forth along one dimension, thus taking more steps to reach the minimum.

If we normialize Instead, if the features are on the same scale, the loss landscape is more uniform like a bowl. Gradient descent can then proceed smoothly down to the minimum.

Feature Normalization:

The image shows two features ( $x_{1}$ and $x_{2}$ ) and how normalization of features can be done before training a machine learning model.

Data in the Image:

The left part shows data where the feature $x_{1}$ has a much larger range than $x_{2}$ . Here, $x_{1}$ spans from 0 to 5, while $x_{2}$ has a smaller range (e.g., 1 to 3).
The goal is to normalize this data, so both features have similar scales, which helps algorithms like gradient descent converge faster.

Subtraction of Mean:

Mean Subtraction: The blue text on the bottom left indicates the calculation for normalizing the data by subtracting the mean:

μ = \frac{1}{m} i = 1 \sum m x^{(i)}

where $m$ is the number of data points, and $x^{(i)}$ is the value of feature $x$ for the $i$ -th data point.

By subtracting the mean $μ$ , we center the data around 0. This helps make the training more stable.

Normalizing the Variance:

Variance Normalization: The blue text on the bottom right indicates the formula for normalizing the variance (standardization):

x^{'} = \frac{x - μ}{σ}

where $σ$ is the standard deviation of the feature, and $x^{'}$ is the normalized feature. This scales the data to have a mean of 0 and a variance of 1.

By normalizing both the mean and the variance, each feature will have a similar scale, making it easier for algorithms to process the data effectively.

Visualizing Normalization:

After normalization, the data will be spread out more evenly along the axes of $x_{1}$ and $x_{2}$ , making the dataset more balanced in terms of the features’ ranges.

Why Normalize Data?

Improves convergence in gradient descent: When features have very different scales, gradient descent might take longer to converge. Normalization helps in faster convergence by giving each feature equal importance.
Prevents bias: Without normalization, features with larger values (e.g., $x_{1}$ ) could dominate the model’s learning process. Normalizing ensures that each feature contributes equally.
Improves performance: Some algorithms (like SVMs, K-means, and neural networks) assume that the data is normalized or standardized. This ensures that the algorithm performs better.

Methods of Normalization:

Standardization (Z-score normalization): Subtract the mean and divide by the standard deviation for each feature.
Min-Max Scaling: Scale the data to a fixed range, usually [0, 1].

Normalization is especially important for models like SVMs, k-NN, and neural networks where the scale of the input features affects model performance.

**Vanishing / Exploding Gradients

The vanishing gradient problem occurs when gradients become very small in deep networks, making it hard for the model to learn. This is often caused by activation functions like sigmoid or tanh, which squash the input into a small range, leading to tiny gradients.

The exploding gradient problem happens when gradients become too large, making the model unstable and causing large updates to the weights, which can lead to poor convergence.

Weight Initialization for Deep Networks

Proper weight initialization is crucial to avoid vanishing/exploding gradients:

Xavier/Glorot Initialization: Initializes weights with a variance based on the number of input and output neurons, which helps prevent exploding or vanishing gradients.
He Initialization: Similar to Xavier, but with a higher variance, better suited for ReLU activation functions.

Gradient Checking

Gradient checking is a method to verify that the gradients computed by your backpropagation algorithm are correct. It compares the gradients from backpropagation with the numerically approximated gradients.

Mini-batch Gradient Descent

Mini-batch gradient descent is a compromise between stochastic gradient descent (SGD) and batch gradient descent. It computes the gradient using a small batch of data instead of the entire dataset or just one sample. This approach offers:

Faster convergence than batch gradient descent.
More stable updates than SGD.

Exponentially Weighted Averages

An exponentially weighted average is a type of moving average that assigns more weight to recent values. It is used in optimization algorithms like Adam to smooth the gradients over time and avoid oscillations.

In optimization, exponentially weighted averages are used for:

Momentum: Helps the gradient descent converge faster by considering past gradients.
Adam: Uses exponentially weighted averages of both gradients and squared gradients to improve convergence

Bias Correction in Exponentially Weighted Averages When using exponentially weighted averages, there’s often a bias towards zero at the beginning of training. Bias correction adjusts the values to account for this, improving the estimates during early training.