Deep learning

 Perceptron

bias →The addition of bias reduces the variance and hence introduces flexibility and better generalisation to the neural network.

Activation Function

• Linear
• Sigmoid or Logistic Activation Function
• Tanh or hyperbolic tangent Activation Function
• ReLU (Rectified Linear Unit) Activation Function ( most used activation function in the world right now)

NOTE: neural nets don’t directly learn curved spirals they learn to transform the input space until straight lines are enough.

Multiple input Preceptron formula

. Simple Line (2D) Equation of a line: $y=mx+c$

• $m$ = slope,
• $c$ = intercept.

This is basically 1 input variable ($x$) and two parameters ($m,c$).

Generalizing to Many Inputs

When we move to multiple inputs ($x_1,x_2,\dots ,x_m$):

$$y=w_1{x}_1+w_2{x}_2+\cdots +w_m{x}_m+w_0$$

• Each input $x_i$ has its own weight $w_i$.
• $w_0$ is the bias (like intercept $c$).

Writing in Matrix Form

Instead of writing long sums, we use vectors & dot products.

• Inputs: $X=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_m\end{array}\right]$

• Weights: $W=\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_m\end{array}\right]$

• Dot product: $X^{T}W=x_1{w}_1+x_2{w}_2+\cdots +x_m{w}_m$

So: $y=w_0+X^{T}W$

Activation Function

The perceptron then applies a non-linearity $g(\cdot )$ (sigmoid, ReLU, etc.):

$\hat{y}=g(w_0+X^{T}W)$

A matrix representation

Every layer in a neural network follows this rule:

$$z^{[l]}=W^{[l]}{a}^{[l-1]}+b^{[l]},a^{[l]}=\sigma (z^{[l]})$$

• $W^{[l]}$: weights for layer $l$
• $b^{[l]}$: biases for layer $l$
• $a^{[l-1]}$: activations from the previous layer
• $a^{[l]}$: activations for this layer (after activation function)
• $\sigma$: activation function (sigmoid, ReLU, etc.)

Linear step (z) + Nonlinear step (σ)

we have:

• Inputs: $x=[x_1,x_2,x_3]$
• Hidden layer (layer 1): 4 neurons
• Output layer (layer 2): 1 neuron

Hidden Layer (Layer 1) Input is $a^{[0]}=x\in {\mathbb{R}}^{(3,1)}$. Formula:

$$z^{[1]}=W^{[1]}{a}^{[0]}+b^{[1]}$$

• $W^{[1]}$: size $(4,3)$ → 4 neurons, each sees 3 inputs
• $a^{[0]}$: size $(3,1)$ → 3 inputs
• $b^{[1]}$: size $(4,1)$
• Result $z^{[1]}$: size $(4,1)$
• Apply activation: $a^{[1]}=\sigma (z^{[1]})(4,1)$

Output Layer (Layer 2)

Input is the hidden layer output $a^{[1]}\in {\mathbb{R}}^{(4,1)}$.

$z^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]}$

• $W^{[2]}$: size $(1,4)$ → 1 output neuron, takes 4 inputs
• $a^{[1]}$: size $(4,1)$
• $b^{[2]}$: size $(1,1)$
• Result $z^{[2]}$: size $(1,1)$
• Apply activation: $a^{[2]}=\sigma (z^{[2]})(1,1)$

This final value $a^{[2]}$ is our prediction $\hat{y}$.

Input (3,1) → Hidden layer (4,1) → Output (1,1)

Full matrix look Input:

$$a^{[0]}=\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$$

Weights:

$$W^{[1]}=\left[\begin{array}{ccc}{w}_{11}& w_{12}& w_{13}\\ w_{21}& w_{22}& w_{23}\\ w_{31}& w_{32}& w_{33}\\ w_{41}& w_{42}& w_{43}\end{array}\right]$$

Bias:

$$b^{[1]}=\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\\ b_4\end{array}\right]$$

Multiply:

$$z^{[1]}=\left[\begin{array}{ccc}{w}_{11}& w_{12}& w_{13}\\ w_{21}& w_{22}& w_{23}\\ w_{31}& w_{32}& w_{33}\\ w_{41}& w_{42}& w_{43}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]+\left[\begin{array}{c}{b}_1\\ b_2\\ b_3\\ b_4\end{array}\right]$$

That gives:

$$z^{[1]}=\left[\begin{array}{c}{w}_{11}{x}_1+w_{12}{x}_2+w_{13}{x}_3+b_1\\ w_{21}{x}_1+w_{22}{x}_2+w_{23}{x}_3+b_2\\ w_{31}{x}_1+w_{32}{x}_2+w_{33}{x}_3+b_3\\ w_{41}{x}_1+w_{42}{x}_2+w_{43}{x}_3+b_4\end{array}\right]$$

NEURONS--MATRIX-REPRESENTATION

• For each input we have different weight in the neurons we can represet the whole layer in single matrix like above z
• After activation: a^[1] = σ(z^[1]) → (4,1)

Note currently we only have single input like x1,x2 ,x3 let say we have 500 feature like 3* 500 we can just convert to matrix and do the parallel computaion

Regularization

Regularization = controls model flexibility so it learns stable, simple, general rules instead of memorizing.

If we dont do the regularization the model will only good at test and train data it will started memorize the data so to avoid we use regularization

Common methods:

1. Weight decay (L2)
2. L1 regularization
3. Dropout
4. Early stopping
5. Data augmentation
6. Batch norm
7. Smaller models
8. Noise injection
9. Label smoothing

check machine learning regulaarization Here

Dropout

To prevent overfitting It works by randomly “dropping out” or deactivating a fraction of the neurons during each training step. This prevents the network from becoming too reliant on specific neurons and forces it to learn more robust features that generalize better to new data.

How Dropout Works:

1. During Training: For each training step, each neuron in the network has a probability p of being dropped out (set to zero). This means the neuron doesn’t contribute to the forward pass or the backward pass (no weight update).
   • For example, if the dropout rate is 0.5, there’s a 50% chance that each neuron will be deactivated in each forward pass.
2. During Testing/Inference: Dropout is turned off during the testing phase. All neurons are active, but their weights are scaled down by a factor of 1−p1 - p1−p (i.e., 50% if p=0.5p = 0.5p=0.5) to ensure that the network does not become too confident about specific features.

Why Use Dropout?

• Prevent Overfitting: Dropout prevents the model from becoming too dependent on any single neuron. Without dropout, some neurons may “memorize” the training data, leading to overfitting. Dropout forces the network to learn redundant representations, making it more robust.
• Improves Generalization: By randomly removing neurons during training, dropout encourages the model to learn more generalized features, improving performance on unseen data.

Example: Imagine we are training a neural network with 4 neurons in a hidden layer, and we apply dropout with a rate of 0.5.

• During training, on each training step, there’s a 50% chance that each neuron will be dropped. So, we might end up with a configuration like this:
  • Step 1: Neurons 1 and 4 are dropped, and the model trains with Neurons 2 and 3.
  • Step 2: Neurons 2 and 3 are dropped, and the model trains with Neurons 1 and 4.

This prevents the network from relying too much on specific neurons and encourages it to generalize better.

Normalization layers (BatchNorm / LayerNorm)

Layer normalization is a technique in deep learning that normalizes the inputs to a layer by computing the mean and standard deviation across all neurons for a single data point. This process stabilizes the training of neural networks, reduces internal covariate shift, and improves convergence by ensuring that the activation values have a consistent distribution.

If we did not do the and we have samll input and big input which make our weights to huge on some parts which is bad.

Normalization forces:

• Mean ≈ 0
• Variance ≈ 1

Which keeps activations from drifting into saturated regions of sigmoid/tanh and keeps ReLU inputs from going too large.

As a result:

• Derivatives for sigmoid/tanh stay in a region where they are not tiny.
• Intermediate activations don’t blow up → Jacobians don’t accumulate large values.

Normalization controls the distribution, stabilizing the product of derivatives.

we do on input data also check

Norimalization formual

$\text{normalized}=\frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}$

Batch Norm

Batch Norm is just another network layer that gets inserted between a hidden layer and the next hidden layer. Its job is to take the outputs from the first hidden layer and normalize them before passing them on as the input of the next hidden layer.

Batch Norm vs layer norm

         feature1   feature2   feature3
sample1:    1          2          3
sample2:    4          5          6
sample3:    7          8          9
sample4:    1          3          5

BN uses ↓      BN uses ↓       BN uses ↓
   [1,4,7,1]     [2,5,8,3]       [3,6,9,5]


sample1 → [1,2,3]
sample2 → [4,5,6]
sample3 → [7,8,9]
sample4 → [1,3,5]

LN uses → only that row

BatchNorm

• Looks across ALL samples
• Normalizes per feature
• Needs big batch
• Behavior depends on mixing examples

LayerNorm

• Looks inside ONE sample only
• Normalizes per sample
• Works with batch size 1
• Used in Transformers, RNNs

BatchNorm compares each feature to the same feature in other samples.
LayerNorm compares each sample to itself.

Gradient Clipping

Gradient clipping is a technique used to prevent the exploding gradients problem during neural network training by scaling down gradients that exceed a certain threshold. This keeps the weight updates from becoming excessively large, which can cause the model to diverge and the training process to become unstable.

Gradient clipping is implemented in two variants:

• Clipping-by-value
• Clipping-by-norm

The idea behind clipping-by-value is simple. We define a minimum and a maximum clip value. 

If a gradient exceeds some threshold value, we clip that gradient to the threshold. If the gradient is less than the lower limit, then we clip that too, to the lower limit of the threshold.

The idea behind clipping-by-norm is similar to clipping-by-value. The difference is that we clip the gradients by multiplying the unit vector of the gradients with the threshold.

Weight intilization

The goal of weight initialization is to keep signals from shrinking or exploding as they pass through layers.

If values shrink → vanishing gradient If values explode → exploding gradient

Method	Why it works	Best for
Random Init	Breaks symmetry but doesn’t control growth/shrink	Not recommended except for small models
Xavier	Sigmoid/tanh compress signals → need balanced weights	Sigmoid, Tanh
He	ReLU kills half activations → needs stronger weights	ReLU, Leaky ReLU
LeCun	SELU self-normalizes → needs smaller weights	SELU
Orthogonal	RNN needs stable repeated multiplication	RNNs

Residual connection

A residual connection, or skip connection, is an architectural innovation in neural networks that bypasses one or more layers to feed the output of an earlier layer directly to a later layer. The core purpose is to enable the training of very deep networks by mitigating problems like vanishing gradients and performance degradation

Resources

• Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift
• Batch Norm Explained Visually – How it works, and why neural networks need it

Exponentially Weighted Average (EWA)

An Exponentially Weighted Average (EWA) is a method used to calculate an average where more recent data points are given more importance (weight) than older ones.

Imagine we are tracking a series of data points over time (like daily temperatures). The goal of an EWA is to smooth out the data and focus on recent trends, while not completely ignoring past data.

• The most recent data point has the highest weight.

• Older data points get less weight.

• Think of it like a weighted average, but the weight of each past value decreases exponentially.

• The further back in time a data point is, the less influence it has on the current average.

So, the recent values matter more, and the older values matter less.

Let’s say we’re tracking the temperature over 5 days:

• Day 1: 20°C
• Day 2: 22°C
• Day 3: 23°C
• Day 4: 25°C
• Day 5: 24°C

If we want to calculate the Exponentially Weighted Average with a smoothing factor $\beta =0.8$, then:

• For Day 1: We start with 20°C (no previous data).
• For Day 2: We take a weighted average of Day 1 and Day 2.
• For Day 3: We take a weighted average of the previous days but give Day 3 the highest weight.

More formally, the formula is:

$$V_t=\beta \cdot V_{t-1}+(1-\beta )\cdot {\Theta }_t$$

Where:

• $V_t$ is the exponentially weighted average on day $t$,
• $\beta$ is the weight given to the previous day’s average (closer to 1 means more weight on the past),
• ${\Theta }_t$ is the temperature on day $t$.

What Does $\beta$ Do?

• $\beta$ = 0.9: More weight is given to past data.
• $\beta$ = 0.1: More weight is given to the most recent data.

If $\beta =0.9$, the most recent temperature will matter a lot, but previous temperatures will still influence the average a little.

If $\beta =0.1$, the most recent temperature will be almost the only thing that matters, and older temperatures won’t influence the average much.

• Noise reduction: It helps smooth out random fluctuations in data.
• React quickly to new trends: Recent data points are more important, so the average can react more quickly to changes.

Resources

• https://towardsdatascience.com/what-is-a-perceptron-basics-of-neural-networks-c4cfea20c590
• https://medium.com/fintechexplained/neural-networks-bias-and-weights-10b53e6285da
• https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0
• https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6
• Deep Learning Interviews is home to hundreds of fully-solved problems, from a wide range of key topics in AI. It is designed to both rehearse interview or exam-specific topics and provide machine learning M.Sc./Ph.D. https://arxiv.org/pdf/2201.00650
• https://deeplearning.neuromatch.io/tutorials/W1D1_BasicsAndPytorch/chapter_title.html

Understanding AI - Lesson 1 / 15: A Simple Neural Network from scrach

• https://www.youtube.com/watch?v=w8n3D8yUZKM&list=PLB0Tybl0UNfYoJE7ZwsBQoDIG4YN9ptyY&index=26

Maths

• The Complete Mathematics of Neural Networks and Deep Learning