Gradient

What is a gradient, and why do we multiply it during training?

• 1 input (x)
• 1 weight (w)
• 1 output
• 1 training example

After this, XOR will be easy.

STEP 1 Define the simplest possible neural network

We take:

$$\hat{y}=w\cdot x$$

One weight (w). One input (x). Output is the multiplication.

Let the training pair be:

$$x=2,y=10$$

We want:

$$w\cdot 2=10\Rightarrow w=5$$

But pretend we don’t know that.

We want training to find this (w).

STEP 2 Define loss function

Take the simplest loss:

$$L=\frac{1}{2}(y-\hat{y}{)}^2$$

This is the function we want to minimize.

Put ( y = w x):

$$L(w)=\frac{1}{2}(10-2w{)}^2$$

The loss depends on one variable: the weight (w).

So learning is simply:

$$\text{Find }w\text{ that makes }L(w)\text{ small.}$$

STEP 3 Gradient = derivative of L(w)

The gradient here is:

$$\frac{dL}{dw}$$

Now compute it:

$$L(w)=\frac{1}{2}(10-2w{)}^2$$

Derivative:

$$\frac{dL}{dw}=(10-2w)(-2)$$

Simplify:

$$\frac{dL}{dw}=-2(10-2w)$$

STEP 4 Ask the ONLY important question:

What does this derivative tell us?

Answer (strict calculus meaning):

$$\frac{dL}{dw}\text{ tells how L changes if w changes slightly.}$$

Example:

Pick an initial weight:

$$w=0$$

Compute derivative:

$$\frac{dL}{dw}{|}_{w=0}=-2(10-2\cdot 0)=-20$$

This value -20 is the entire key.

Let’s understand only this one number.

STEP 5 What does “-20” actually mean? (no intuition)

Plug back into calculus meaning:

$$\frac{dL}{dw}\approx \frac{\Delta L}{\Delta w}$$

So:

$$-20\approx \frac{\Delta L}{\Delta w}$$

This says:

If w increases by 1, L decreases by roughly 20.

That is: Increase (w) → Loss goes down.

If instead we decreased (w), L would go up.

This comes directly from the sign.

So the derivative contains TWO numbers:

1. Sign (increase/decrease)
2. Magnitude (how strong)

That’s ALL we need to update the weight.

STEP 6 Why do we multiply the gradient?

The update rule is:

$$w_{\text{new}}=w_{\text{old}}-\eta \frac{dL}{dw}$$

Let’s plug the numbers:

• (\eta = 0.1)
• (\frac{dL}{dw} = -20)

Update:

$$w_{\text{new}}=0-0.1(-20)$$ $$w_{\text{new}}=2$$

Now compute the new loss:

Old loss:

$$L(0)=\frac{1}{2}(10-0{)}^2=50$$

New loss:

$$L(2)=\frac{1}{2}(10-4{)}^2=18$$

Loss dropped from 50 to 18.

Why? Because the gradient told us exactly how L changes when w changes.

That’s all training is.

STEP 7 One more update

Use (w = 2).

Compute derivative:

$$\frac{dL}{dw}{|}_{w=2}=-2(10-4)=-12$$

Do update:

$$w_{\text{new}}=2-0.1(-12)$$ $$w_{\text{new}}=3.2$$

Loss now:

$$L(3.2)=\frac{1}{2}(10-6.4{)}^2=6.48$$

Loss keeps dropping.

STEP 8 Continue until w ≈ 5

Gradient descent will move:

$$w=0\to 2\to 3.2\to 4.16\to 4.73\to 4.94\to \dots$$

Eventually:

$$w\approx 5$$

which makes:

$$\hat{y}=2w=10$$

• Loss (L(w)) is a function.
• Training = minimizing this function.
• The gradient is the exact rate of change of L with respect to w.
• If gradient = -20, it means increase w → L decreases fast.
• Update rule subtracts gradient to move L downward.
• Step size controlled by (\eta).
• Repeating updates finds the value that minimizes loss.

Everything else in neural networks is just the same process with more weights and chain rule.