Algebra Vectors & matrics

├── Linear Algebra
│   ├── Scalars
│   ├── Vectors
│   │   ├── Vector Addition
│   │   ├── Vector Norms
│   │   │   ├── L1 Norm
│   │   │   ├── L2 Norm
│   │   │   └── Max Norm (L∞)
│   │   ├── Dot Product
│   │   ├── Cross Product                          
│   │   ├── Cosine Similarity
│   │   ├── Orthogonality                           
│   │   └── Projection                              
│   │
│   ├── Matrices
│   │   ├── Matrix Addition
│   │   ├── Matrix Multiplication
│   │   ├── Transpose
│   │   ├── Inverse
│   │   ├── Determinant
│   │   ├── Trace                                   
│   │   ├── Matrix Norms (Frobenius, Spectral)      
│   │   ├── Symmetric Matrices                      
│   │   ├── Positive Definite Matrices              
│   │   ├── Orthogonal Matrices                     
│   │   └── Sparse Matrices                         
│   │
│   ├── Tensors                                     
│   │   ├── Tensor Operations                       
│   │   └── Tensor Decomposition                    
│   │
│   ├── Linear Transformations
│   ├── Span and Linear Independence                
│   ├── Basis and Dimension                         
│   ├── Eigenvalues
│   ├── Eigenvectors
│   ├── Diagonalization
│   ├── Singular Value Decomposition (SVD)
│   ├── Rank
│   └── Pseudoinverse (Moore-Penrose)               

Straight line

The slope tells us how many units we move vertically (up or down) for every unit we move horizontally (right or left).

y = mx+b

m = x2-x1/y2-y1

Vector

A vector is more than just an arrow on a graph. It’s a container of information it can hold direction, position, or a list of attributes describing a real-world object.

Imagine you’re evaluating houses. Each house has:

• Floor area (in sq. ft)
• Number of bedrooms
• Number of bathrooms
• Price (in thousands)

You can represent a house as a vector:

h = [
		1800
		30
		2
		230
	]

Now, all houses in your dataset become points in 4D space, where each axis is an attribute. This way, you can compare houses geometrically e.g., how “far apart” they are in feature space.

In physics, a vector might represent:

• Velocity: 30 m/s north
• Force: 100 N at 45°

So vectors aren’t just lists they mean something. A force vector tells you how hard and in which direction something is being pushed.

Matrices Linear Transformations

Matrices are machines that act on vectors to transform them. They rotate, scale, flip, or skew the vector space.

Example : When we apply a filter, it alters the brightness, contrast, and color tones.

What’s happening under the hood? The pixels are vectors of RGB values, and the filter is a matrix transformation:

Basis and Coordinate Systems

Vector basis

Assume we have unit vector of 1 unit of x Axis and Y axis if we multiple the unit vector with 2 and 3 we get our vector v

So every vector we can represent by using unit vector which is called vector basis

We have choosed the x1 and y1 as basis but we can choose any basis as base for vector

Vector concepts

Vector length (norm / magnitude)

Euclidean distance from origin to the vector’s tip

If a vector is →v = (3, 4), then draw it from the origin to the point (3, 4).
That forms a right triangle with sides 3 and 4.

The formula for distance between two points was

$$\sqrt{(x_1-x_2{)}^2+(y_1-y_2{)}^2}$$

For our case x2 and y2 are 0 because it is in origin so

$$\Vert \vec{v}\Vert =\sqrt{(v_1-0{)}^2+(v_2-0{)}^2}=\sqrt{v_1^2+v_2^2}$$

Norm vector

A normal vector (or norm vector) is a vector that is perpendicular to a given surface, line, or plane.

In the context of 2D geometry, for a line in the form $ax+by+c=0$, the normal vector is a vector that points perpendicular to the line. This vector is defined by the coefficients of $x$ and $y$ in the line’s equation.

For the line $ax+by+c=0$:

Same the equation of line represent in slope form is y = mx+b

The normal vector is: $\vec{n}=(a,b)$

Why is it called a “normal” vector?

• The word “normal” comes from the mathematical term “normal to,” meaning “perpendicular.”
• A normal vector is perpendicular to the line (or plane, in 3D) it is associated with.

How is it used?

The normal vector $\vec{n}=(a,b)$ tells you the direction of the line’s “steepness” or “slope” and is essential for various calculations, such as:

• Finding the shortest distance from a point to a line
• Defining the orientation of the line.

Consider the line $2x+3y-6=0$:

• The normal vector is $\vec{n}=(2,3)$.
• This vector is perpendicular to the line at every point, and it provides the direction you would travel to get from the line in a straight path.

How to find Normal vector?

Let’s say you have a point $(x,y)$ that lies on the line. For this point to satisfy the equation of the line, when you plug $x$ and $y$ into the equation, it must result in 0:

$$ax+by+c=0$$

This means that the point $(x,y)$ satisfies the line’s equation.

Vectors and Dot Product:

Now, let’s think about this in terms of vectors.

Position Vector of the Point:

The position vector $\vec{v}$ of the point $(x,y)$ is simply the vector from the origin to the point. This is written as: $\vec{v}=(x,y)$

Normal Vector:

Now, the normal vector to the line is defined as $\vec{n}=(a,b)$, where $a$ and $b$ are the coefficients from the line’s equation. The normal vector is always perpendicular to the line, which means that it points in the direction that is orthogonal (at a 90° angle) to any vector lying on the line.

Why the Dot Product is Zero: The dot product of two vectors $\vec{v}=(x,y)$ and $\vec{n}=(a,b)$ is given by: $\vec{v}\cdot \vec{n}=ax+by$

Here’s the key idea:

• For two vectors to be perpendicular, their dot product must be zero. This is because the dot product measures how much one vector “projects” onto the other. If the vectors are perpendicular, there’s no projection, so the dot product is zero.

In our case:

• If the point $(x,y)$ lies on the line, then the equation $ax+by+c=0$ is satisfied.

• This implies that the dot product $\vec{v}\cdot \vec{n}=ax+by$ equals $-c$.

• So the dot product gives you a measure of the distance from the origin to the point, but it doesn’t affect the fact that the vector $\vec{n}=(a,b)$ is perpendicular to any vector along the line.

Thus, $\vec{n}$ is normal to the line because the dot product of the normal vector $\vec{n}=(a,b)$ with any vector lying along the line will always be zero.

Dot product

A scalar value that tells us how much one vector goes in the direction of another

$$\vec{v}\cdot \vec{w}=v_1{w}_1+v_2{w}_2$$

How much of vector w⃗ lies in the direction of v?

Depending on the sign of the result:

• Positive → Vectors point roughly in the same direction
• Zero → Vectors are perpendicular (orthogonal)
• Negative → Vectors point in opposite directions

• Blue vector = u, Green vector = v.
• The red dashed line is the actual distance ∥u−v∥ between them.
• The dot product u⋅vu \cdot vu⋅v is not this red length. Instead, it measures how much u “falls” in the direction of v(a projection).

Example

Imagine we trying to predict whether a movie will be liked by a user based on the user’s preferences (represented by a vector) and the movie’s attributes (e.g., genre, director, actors, etc., also represented as a vector). The dot product between the user’s preference vector and the movie’s feature vector can tell you how much the movie “aligns” with the user’s preferences. If the dot product is large, the movie is likely to be liked; if it’s small or negative, the movie is less likely to be liked.

• Movie vector: $[8,9,7]$ (representing Action genre, Christopher Nolan, Tom Hardy)
• User vector: $[6,10,4]$ (representing moderate Action interest, love for Nolan, dislike of Tom Hardy)
• Dot product: 166 (tells us how well this movie matches the user’s preferences).

To get the threshold like if dot product above 150 user like etc.

calculate the threshold based on historical data. The idea is to set the threshold so that it accurately predicts whether users will like or dislike a movie based on their past behavior

Angle between two vectors

Introduced via the cosine formula in the dot product

$$\vec{v}\cdot \vec{w}=\Vert \vec{v}\Vert \cdot \Vert \vec{w}\Vert \cdot \cos (\theta )$$ $$\cos (\theta )=\frac{\vec{v}\cdot \vec{w}}{\Vert \vec{v}\Vert \cdot \Vert \vec{w}\Vert }$$

Projection of one vector onto another

The component of one vector that “falls” onto the line of the other

• “How much of A points in the direction of B?”
• Its length is how much A “lies along” B.
• The difference between A and its projection is perpendicular to B.

Linear Combination

A linear combination means we’re creating new vectors by scaling and adding existing ones.

we have unit vector which can be represent all other vector

$$\vec{i}=\left[\begin{array}{c}1\\ 0\end{array}\right],\vec{j}=\left[\begin{array}{c}0\\ 1\end{array}\right]$$

Assume we have two number 2 and 3 that is alpha and beta if we muliptly we get a vector

$$\vec{v}=\alpha \cdot \vec{i}+\beta \cdot \vec{j},\alpha ,\beta \in \mathbb{R}$$ $$\vec{v}=\left[\begin{array}{c}2\\ 3\end{array}\right]=2\cdot \vec{i}+3\cdot \vec{j}$$

Linearly independent

• Linearly independent vectors are like arrows pointing in different directions, so none of them can be created by stretching or combining the others.

• If you cannot write one vector as a combination (stretching + adding) of the others, then those vectors are linearly independent.

• Vectors are linearly dependent if at least one vector can be made by stretching and adding the others.

$$\vec{w}=\left[\begin{array}{c}1\\ 1\end{array}\right],\vec{z}=\left[\begin{array}{c}2\\ 2\end{array}\right]$$

we can represent the z vector by w vector if we multiply w by 2

Note: If vector is linearly dependent mean there is reduncey in vector we can remove unwanted vector.

Span

The span of vectors is the set of all possible linear combinations of those vectors.

Span is nothing but we can create any no of vector by replacing the alpha and beta in our linear combination that what span

$$\vec{i}=\left[\begin{array}{c}1\\ 0\end{array}\right],\vec{j}=\left[\begin{array}{c}0\\ 1\end{array}\right]$$

if we multiple by our scaler alpha and beta of i and j we get new vector if keep changeing the alpha and beta we get new points

We can multiply any number we get points if we map all points in graph we can represent anywhere in the map

minimal spanning set is:

• A spanning set with no unnecessary (redundant) vectors.
  • You already can reach all points in the space without it.
  • It’s not giving you any new direction.
  • Example {(1,0),(0,1),(1,1)}
  • This means (1,1) is a linear combination of the other two.
  • If you remove (1,1), you still have the full plane → the span is unchanged.
  • ❌ Redundant vector → Not minimal.
  • So this is not basis
• If you remove any vector from it, it no longer spans the space.

In other words: You keep the set as small as possible while still covering the whole space.

basis

A basis is a minimal set of independent vectors that can span a space. our bias for 2D

$$\vec{i}=\left[\begin{array}{c}1\\ 0\end{array}\right],\vec{j}=\left[\begin{array}{c}0\\ 1\end{array}\right]$$

Let take example we have vector of data for rooms so we have vector for no of bedroom and bathroom represent as

x - 2y = 0  
2x - 4y = 0



second example where they not crossed over origin
x - 2y = 3  
2x - 4y = 6

If we plot that in graph we get two same line both are linearly dependent

If we solve the equation

x= 2y

so if we take what every x y is double then x that what it trying to say so the biasis for this 2,1

A basis vector like 2,1 is the smallest, simplest arrow that points exactly along the solution line.

We can able to plot all the data points in line by using 2,1

Every other solution is just that arrow, stretched or flipped.

Null space and kernel

A linear transformation of a matrix take in to the zero of the origin If you have a matrix A and you solve:

$$Ax=0$$

Any solution X you find is a null space vector.

Those vectors are special because:

• They get completely erased by the transformation A they become the zero vector.
• Geometrically, they’re the directions that get squashed to a single point at the origin.

If we mulitply (linear transformation) by vector which give us zero which is called null or kernel vector

Vector vs Matrix

• A vector can represent a single data point with multiple features (or dimensions). For example, a vector represents one data point with nnn features.

• A matrix can represent multiple data points, each having multiple features. For example, an m×nm \times nm×n matrix represents mmm data points (rows), each with nnn features (columns).

A matrix is a rectangular arrangement of numbers, symbols, or functions organized in rows and columns. Its size (or order) is given by the number of rows m and columns n. We denote this as:

$$A_{m\times n}$$

For example, a 3×3 matrix looks like this:

$$A_{3\times 3}=\left[\begin{array}{ccc}1& 4& 12\\ 9& 5& 1\\ 8& 2& 7\end{array}\right]$$

More generally, a matrix can be represented as:

$$A_{m\times n}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right]$$

Matrices in Machine Learning

In machine learning, matrices are often used to organize features measurable properties or characteristics of the data for multiple samples.

Suppose we want to predict whether students will make the basketball team based on three features:

• Height (in meters)
• Weight (in kilograms)
• Average grade (on some scale)

Each student is represented as a vector of features:

$${\text{Student}}_1=(1.65,64,5)$$ $${\text{Student}}_2=(1.54,67,4)$$ $$⋮$$ $${\text{Student}}_n=(1.61,55,2)$$

Combining all students’ vectors forms a feature matrix XX:

$$X=\left[\begin{array}{ccc}1.65& 64& 5\\ 1.54& 67& 4\\ ⋮& ⋮& ⋮\\ 1.61& 55& 2\end{array}\right]$$

Alongside this matrix, you might have a target vector yy indicating if each student made the team:

$$y=\left[\begin{array}{c}\text{yes}\\ \text{yes}\\ ⋮\\ \text{no}\end{array}\right]$$

This dataset (feature matrix + target vector) is then used to train machine learning models.

Linear Transformation

A linear transformation is a rule (or function) that takes a vector as input and gives another vector as output in a way that preserves straight lines and scales/proportions.

But need stasify these rules

• Additivity: T(u+v)=T(u)+T(v)
•  Homogeneity:T(c⋅v)=c⋅T(v)

What does a linear transformation look like?

• Stretching
• Rotating
• Reflecting
• Shearing (slanting)
• Compressing
• Identity (does nothing)

But no:

• Bending curves
• Shifting (translation)
• Warping

Determinant

Is the area between two vector reprsent as in matrix

a b 
c d 

$$\text{Area}=|ad-bc|$$

How the formula came

(a + b)(c + d) − ac − bd − 2bc = ad − bc

Let’s take:

• v₁ = (2, 1)
• v₂ = (3, 4)

Then:

Area=∣2×4−1×3∣= ∣8−3∣ =5

That means these two vectors span a parallelogram with area 5.

We can compute area before and after linear transformation and it tell

• Determinant = 1 → area stays the same
• Determinant = 2 → area doubles
• Determinant = 0 → area is 0 → the transformation collapsed everything into a line (no area) also they are linearly dependent vector

Column space

The only building blocks for outputs are the columns of the matrix.

$$A=\left[\begin{array}{cc}1& 2\\ 0& -1\\ 3& 0\end{array}\right]$$

Columns:

$${\mathbf{c}}_1=\left[\begin{array}{c}1\\ 0\\ 3\end{array}\right],{\mathbf{c}}_2=\left[\begin{array}{c}2\\ -1\\ 0\end{array}\right]$$

If we feed in (x,y) to A

$$A\left[\begin{array}{c}x\\ y\end{array}\right]=x{\mathbf{c}}_1+y{\mathbf{c}}_2=x\left[\begin{array}{c}1\\ 0\\ 3\end{array}\right]+y\left[\begin{array}{c}2\\ -1\\ 0\end{array}\right]=\left[\begin{array}{c}x+2y\\ -y\\ 3x\end{array}\right]$$

So every output is some combination of these two columns.

That’s the column space:

• All vectors you can make by combining the columns.

$$\text{Col}(A)=\left\{A\mathbf{x}\mid \mathbf{x}\in {\mathbb{R}}^n\right\}=\text{span}\left\{{\mathbf{c}}_1,{\mathbf{c}}_2\right\}$$

• If c1 and c2 are independent → column space is a plane in ${\mathbb{R}}^3$
• If they are dependent (one is multiple of the other) → column space is a line in ${\mathbb{R}}^3$

Rank

The number of independent columns (or rows it’s the same).

• Rank = dimension of the column space.
• Rank tells you how many independent directions the matrix can produce.

Rank tell how it look in graph does it was 2D or 3D etc

let say if we have matrix

1 2
2 3 

IF we multiply by any vector x,y we able to cover all space in 2D plane because both are linearly independent so the RANK is 2

1 2
3 6

Here they are linearly dependent where they have only line if multiply by any vector it will be on line only

Methods to find rank

There are 3 common ways:

1. Row-reduction (Gaussian elimination) → Most practical by hand.
2. Determinants → Works for square matrices (check minors).
3. Singular Value Decomposition (SVD) → Used in computers.

The application of RANK help us to reduce the size of image let say we have image with matrix 264*264 l

The rank of a matrix tells you how many independent rows (or columns) it has.

• If rank = no of rows, all rows/columns are independent → you need all of them to represent the data exactly.
• If rank is smaller, the rest of the rows/columns can be expressed as combinations of a few independent ones.

If most of an image’s matrix can be reconstructed from a few independent components, we can store only those and throw away the rest reducing size.

If one row is exactly the average of two other rows, we don’t need to store it — we can just say:

If one row is exactly the average of two other rows, we don’t need to store it — we can just say:

$$\text{Row 3}=0.5\times \text{Row 1}+0.5\times \text{Row 2}$$

Null space

If we multiply a matrix by any vector it will always give 0

Invertablity

A matrix is invertible if we can “undo” the transformation it performs.

Just like how the inverse of a number “undoes” its multiplication:

5 * 1/5 = 1 

For a matrix A, if there exists another matrix

$$A^{-1}$$ $$A{A}^{-1}=I$$

where I is the identity matrix), then A is invertible.

• If det⁡(A)≠0, then the matrix is invertible.
• If det⁡(A)=0, then the matrix is not invertible.

Matrix	Effect	Determinant	Sign Meaning
Scaling only	Expands or shrinks space	>0	Preserves orientation
Rotation	Just rotates space	>0	Preserves orientation
Reflection	Flips like a mirror	<0	Reverses orientation
Degenerate	Squashes space (flat)	=0	Loses dimension

A matrix has an inverse only if:

• It is square (same number of rows and columns), and
• Its determinant is not zero

systems of linear equations

A system of linear equations is just a way to express relationships like:

$$\{\begin{array}{l}2x+3y=8\\ x-4y=-2\end{array}$$

We want to find the values of x and y that satisfy both equations at the same time.

$$A\vec{x}=\vec{b}$$ $$A=\left[\begin{array}{cc}2& 3\\ 1& -4\end{array}\right]\vec{x}=\left[\begin{array}{c}x\\ y\end{array}\right]\vec{b}=\left[\begin{array}{c}8\\ -2\end{array}\right]$$

A- the coefficient matrix

we can now use matrix techniques (like inverse matrices) to solve the system all at once.

$$\vec{x}=A^{-1}\vec{b}$$

This means we just compute the inverse of A, multiply by b⃗ and we get your solution

• Homogeneous: Geometrically, solutions form a subspace (passes through the origin). Ax=0

-2x+y−3z=0−x+4y+z=0

 Homogeneous equations always have at least one solution the trivial solution
x=0,y=0,z=0

• Non-homogeneous: Solutions form an affine space (shifted, does not necessarily pass through origin). Ax=b

2x+y−3z=5−x+4y+z=−2

Eigenvectors

Imagine we apply a transformation (matrix) to a vector. Most vectors change direction and length.

But some very special vectors only change length, not direction.

Those are called eigenvectors.

And the amount they stretch/shrink by is called the eigenvalue.

If we multiply a matric by a vector v we get some output but we can take the common scale from the output which again give our vector

Example

M  = 2 1
	 1 2 

vector = 1 
		 1 

if we multiply M * v =  3 
						3  

But if we take 3 outside we again get the vector v 3 = 1
														1


so the eigenvector is 1,1 and eigenvalue is 3 

That what telling in formula 
	A * v = lambda * vector -> which eignvector and vaule

Av=λv

Where:

• A is the transformation matrix,
• v is the eigenvector (must be non-zero),
• λ is the eigenvalue (a scalar).

It means:

“When we apply A to v⃗, we just scale it by λ— no rotation.”

If we solve the equation we get formula to find eigenvaules and vector

Av−λv=0

Factor out v

(A−λI)v=0

We now want non-zero v⃗ to satisfy this, which is only possible if:

det(A−λI)=0 

This gives us a characteristic polynomial in λ, whose roots are the eigenvalues.

Indepth

Eigenvectors make understanding linear transformations easy. They are the ‘axes’ (directions) along which a linear transformation acts simply by ‘stretching/compressing’ and/or ‘flipping’; eigenvalues give you the factors by which this compression occurs.

Matrix Diagonalization

Matrix Diagonalization is the process of rewriting a matrix in a simpler, diagonal form using its eigenvectors and eigenvalues.

Matrix diagonalization is the process of expressing a matrix A as:

A=PDP^−1

• D is a diagonal matrix with the eigenvalues of A on the diagonal.
• P is a matrix with the eigenvectors of A as its columns.

Diagonalization is like changing the basis of a matrix so it becomes simpler (diagonal), allowing:

• Easy computation of powers
• Insight into matrix structure
• Application to physics, ML, control systems

To find eigenvalues by solving the characteristic equation:

det(A−λI)=0

• Detereminted * a - lambda * I = 0

Orthogonal

Two vectors are orthogonal if their dot product is zero:

$$\mathbf{u}\cdot \mathbf{v}=0$$

This means they’re perpendicular and independent.

A vector is normal (in this context) if its length is exactly 1:

$$\Vert \mathbf{u}\Vert =1$$

This is called a unit vector — it keeps only the direction, not extra size.

A set of vectors is orthonormal if:

1. Orthogonal to each other

$${\mathbf{u}}_i\cdot {\mathbf{u}}_j=0\text{for}i\ne j$$

2. Normal (unit length) individually

$$\Vert {\mathbf{u}}_i\Vert =1$$

So:

• Orthogonal = perpendicular, but can be any length.
• Orthonormal = perpendicular and each one is length 1.

Orthogonal but not orthonormal:

$$\mathbf{u}=(3,0),\mathbf{v}=(0,2)$$

Dot product is $0$, but lengths are $3$ and $2$. Orthonormal version (divide each by its length):

$${\mathbf{u}}^{\prime }=(1,0),{\mathbf{v}}^{\prime }=(0,1)$$

Dot product is still $0$, but now lengths are $1$.

Matrix Decomposition

Matrix Decomposition (or Factorization) means expressing a matrix as a product of simpler matrices that are easier to analyze or compute with.

• break a number like 606060 into 2×2×3×52
• break a function like sin⁡(x)\sin(x)sin(x) into a Taylor series
• break a sound into frequencies (via Fourier transform)

Name	Works On	Purpose
LU Decomposition	Square matrices	Efficiently solve linear systems Ax=bAx = b
QR Decomposition	Any matrix	Used in least squares, numerical stability
Cholesky Decomposition	Symmetric, pos-def	Faster version of LU for specific matrices
Eigen Decomposition	Square matrices	Diagonalize matrices (if possible)
SVD (Singular Value)	All matrices (any size)	General-purpose, powerful for data + compression

PCA

Mean

The mean (often called the average) measures the central tendency — where the “middle” of the data lies.

Formula for the mean of $n$ values $x_1,x_2,\dots ,x_n$:

$$\mu =\frac{x_1+x_2+\cdots +x_n}{n}$$

If data is: $2,4,6,8$

$$\mu =\frac{2+4+6+8}{4}=\frac{20}{4}=5$$

This tells you that the “center” of the numbers is at 5.

we have different 8 image where we get mean by adding all pixel and divide by no of image we get mean

Variance?

Variance measures spread how far the numbers are from the mean. Formula for population variance: (How far is one variable from its center?)

$${\sigma }^2=\frac{(x_1-\mu {)}^2+(x_2-\mu {)}^2+\cdots +(x_n-\mu {)}^2}{n}$$

Example (same data): $2,4,6,8$, mean $\mu =5$

1. Differences from mean: $2-5=-3$, $4-5=-1$, $6-5=1$, $8-5=3$
2. Squares: $9,1,1,9$
3. Average:

$${\sigma }^2=\frac{9+1+1+9}{4}=\frac{20}{4}=5$$

Variance here = 5.

CoVariance

Covariance is a statistical measure of how two random variables change together

• If they tend to increase together, the covariance is positive.
• If one increases while the other decreases, the covariance is negative.
• If they are unrelated, the covariance is close to zero.

Covariance tells us the direction of a relationship but not the strength. (That’s what correlation does more on that later.)

Formula of Covariance

For two random variables $X$ and $Y$, the population covariance is:

$$\text{Cov}(X,Y)=\frac{1}{n}\sum _{i=1}^n(X_i-{\mu }_X)(Y_i-{\mu }_Y)$$

• $X_i,Y_i$: the ith data points
• ${\mu }_X,{\mu }_Y$: means of X and Y
• $n$: number of data points

For the sample covariance, we divide by $n-1$ instead of $n$:

$$\text{Cov}(X,Y)=\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$$

What Does Covariance Really Mean?

Let’s break down this formula:

• $(X_i-\bar{X})$ tells us how far $X_i$ is from the mean of $X$.

• $(Y_i-\bar{Y})$ tells us the same for $Y$.

• Multiply them → if both deviations are positive or both are negative, the result is positive (they move together).

• If one is positive and the other is negative → result is negative (they move in opposite directions).

Then, take the average of all those products. That’s the covariance.

Example:

Let’s say we have data for two variables:

Student	Hours Studied (X)	Exam Score (Y)
A	2	65
B	4	70
C	6	75
D	8	85
E	10	95

Step 1: Calculate the Means

• $\bar{X}=(2+4+6+8+10)/5=6$
• $\bar{Y}=(65+70+75+85+95)/5=78$

Step 2: Use Formula

Now compute each term $(X_i-\bar{X})(Y_i-\bar{Y})$:

X	Y	X−XˉX - \bar{X}	Y−YˉY - \bar{Y}	Product
2	65	-4	-13	52
4	70	-2	-8	16
6	75	0	-3	0
8	85	2	7	14
10	95	4	17	68

Sum of products = $52+16+0+14+68=150$

Sample size = 5 → use sample covariance (divide by 4):

$$\text{Cov}(X,Y)=\frac{150}{4}=37.5$$

It’s a positive number, so:

• As hours studied increases, exam score tends to increase.
• The number 37.5 itself doesn’t say how strong the relationship is, just that they’re positively related.

Tools

• https://www.geogebra.org/3d?lang=en
• https://www.desmos.com/calculator