Machine learning data

normalizing training sets

When inputting data to a deep learning model, it is standard practice to normalize the data to zero mean and unit variance. What does this mean and why do we do this?

Let’s say the input data consists of several features x1, x2,…xn. Each feature might have a different range of values. For instance, values for feature x1 might range from 1 through 5, while values for feature x2 might range from 1000 to 99999.

So, for each feature column separately, we take the values of all samples in the dataset and compute the mean and the variance. And then normalize the values using the formula below.

To understand what happens without normalization, let’s look at an example with just two features that are on drastically different scales. Since the network output is a linear combination of each feature vector, this means that the network learns weights for each feature that are also on different scales. Otherwise, the large feature will simply drown out the small feature.

Then during gradient descent, in order to “move the needle” for the Loss, the network would have to make a large update to one weight compared to the other weight. This can cause the gradient descent trajectory to oscillate back and forth along one dimension, thus taking more steps to reach the minimum.

If we normialize Instead, if the features are on the same scale, the loss landscape is more uniform like a bowl. Gradient descent can then proceed smoothly down to the minimum.

Feature Normalization:

The image shows two features ($x_1$ and $x_2$) and how normalization of features can be done before training a machine learning model.

2. Data in the Image:

• The left part shows data where the feature $x_1$ has a much larger range than $x_2$. Here, $x_1$ spans from 0 to 5, while $x_2$ has a smaller range (e.g., 1 to 3).
• The goal is to normalize this data, so both features have similar scales, which helps algorithms like gradient descent converge faster.

2. Subtraction of Mean:

• Mean Subtraction: The blue text on the bottom left indicates the calculation for normalizing the data by subtracting the mean:

$$\mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)}$$

where $m$ is the number of data points, and $x^{(i)}$ is the value of feature $x$ for the $i$-th data point.

• By subtracting the mean $\mu$, we center the data around 0. This helps make the training more stable.

Normalizing the Variance:

• Variance Normalization: The blue text on the bottom right indicates the formula for normalizing the variance (standardization):

$$x^{\prime }=\frac{x-\mu }{\sigma }$$

where $\sigma$ is the standard deviation of the feature, and $x^{\prime }$ is the normalized feature. This scales the data to have a mean of 0 and a variance of 1.

• By normalizing both the mean and the variance, each feature will have a similar scale, making it easier for algorithms to process the data effectively.

Visualizing Normalization:

• After normalization, the data will be spread out more evenly along the axes of $x_1$ and $x_2$, making the dataset more balanced in terms of the features’ ranges.

Why Normalize Data?

• Improves convergence in gradient descent: When features have very different scales, gradient descent might take longer to converge. Normalization helps in faster convergence by giving each feature equal importance.

• Prevents bias: Without normalization, features with larger values (e.g., $x_1$) could dominate the model’s learning process. Normalizing ensures that each feature contributes equally.

• Improves performance: Some algorithms (like SVMs, K-means, and neural networks) assume that the data is normalized or standardized. This ensures that the algorithm performs better.

Methods of Normalization:

• Standardization (Z-score normalization): Subtract the mean and divide by the standard deviation for each feature.
• Min-Max Scaling: Scale the data to a fixed range, usually [0, 1].

Normalization is especially important for models like SVMs, k-NN, and neural networks where the scale of the input features affects model performance.

Vanishing gradients, initialization, and averages

Vanishing and exploding gradients, weight initialization, gradient checking, mini-batch gradient descent, and exponentially weighted averages are in Activation functions.