Machine Learning Strategy

Machine Learning (ML) strategy involves aligning your model’s design, data handling, and optimization processes to meet specific goals effectively. A clear strategy ensures efficient workflow and better model outcomes.

Orthogonalization

Orthogonalization refers to making sure that different features (or models) contribute independently and don’t interfere with each other. This helps reduce redundancy and improves model performance.

Analysis of Multi-Metric Problems: Often, performance is measured using multiple metrics that conflict, making decision-making difficult.

1. Precision and Recall: If Classifier A has 90% recall but 95% precision, and Classifier B has better precision but worse recall, it is difficult to determine which is superior.

2. Geographical Performance: If an algorithm yields different error rates across four major geographies (US, China, India, Other), you have four numbers to track, making rapid selection impossible.

Solution: Combining Metrics: To resolve this, metrics should be combined:

1. F1 Score: Precision and recall are commonly combined using the F1 score, which is the harmonic mean of the two. Using the F1 score allows immediate selection of the better classifier (e.g., Classifier A).

2. Averaging: For errors across different geographies, simply computing the average performance provides a single number to quickly identify the best algorithm (e.g., Algorithm C

Using a single metric (like accuracy, F1-score, or AUC) helps measure your model’s performance in a way that is easy to track and compare. This simplifies decision-making and goal-setting in your ML workflow.

Error Analysis and Data Labeling

Manual Error Analysis: The process of manually examining a subset of mislabeled examples from the dev set to identify and count categories of mistakes. This quickly prioritizes which problems (e.g., dogs, blurry images) are the biggest bottlenecks.

Ceiling on Performance: Error analysis provides an upper bound on how much improvement can be gained by fixing a specific type of error.

Incorrectly Labeled Examples: Refers to errors in the output labels (Y values) within the dataset itself.

Robustness to Training Set Errors: Deep learning algorithms are generally robust to random errors in the training set but less robust to systematic errors.

Correcting Labels in Dev/Test Sets: Fixing incorrect labels in the dev and test sets is recommended if the percentage of label errors constitutes a significant fraction of the overall dev set error

Bias and variance intiuion

Why would more data sometimes help a model, and sometimes not help at all?

To answer this from first principles, we need a mechanistic view of what “bias” and “variance” are inside a learning system.

2. Constructing the mental model

A model tries to approximate some true function

Call the true mapping:

$$y=f(x)$$

Your model produces:

$$\hat{y}=g(x;\theta )$$

All training no matter the architecture is trying to make:

$$g(x;\theta )\approx f(x)$$

There are two possible failure modes:

3. Mechanistic meaning of HIGH BIAS

A model has high bias when its function family is too limited to approximate the true function, even with perfect training.

Mechanistically:

• The model class ( g(x) ) is too simple.
• The gradients cannot move (\theta) into a region that represents the true mapping.
• Even with infinite data, the “best possible fit” is still far from the true function.

Example mechanism:

A linear model trying to represent XOR.

No matter how many examples you add, the hypothesis class cannot bend enough to match the curved decision boundary. You are stuck with:

${\min }_{\theta }\mathbb{E}[(g(x;\theta )-f(x){)}^2]\text{still large}$

Adding data does not change the shape of the model. So the error stays high.

Therefore: More data cannot fix bias, because the problem is structural.

Mechanistic meaning of HIGH VARIANCE

A model has high variance when its function family is flexible enough but the parameters fluctuate too much depending on the specific samples you train on.

Mechanistically:

• The model can express many different functions.
• The gradient descent solution depends heavily on which samples it sees.
• With limited data, weight updates follow noise patterns in the training set.

In equations:

$${\theta }_{\text{train 1}}\ne {\theta }_{\text{train 2}}$$

Even though both sets come from the same distribution.

This means the model is overfitting—it captures patterns that are not stable across datasets.

Now ask:

How does more data help?

With more samples:

• Random noise patterns average out.
• Expected gradient becomes closer to true gradient of population loss.
• Weight updates converge to a more stable region of parameter space.

Therefore, variance reduces as dataset size grows.

Mathematically, for many models:

$$\text{Variance}\propto \frac{1}{N}$$

where (N) is dataset size.

“If a learning algorithm suffers from high bias, more training data won’t help.”

Because:

• The model shape cannot represent the underlying function.
• The error comes from structural limitation.
• More data just reinforces the same wrong function.

“If a learning algorithm suffers from high variance, more data will help.”

Because:

• The model can represent the correct function.
• But it currently overfits due to insufficient samples.
• More samples stabilize the gradient and tighten the estimate.

High Bias

Model is blind. The body of the model cannot bend into the needed shape. Feeding more examples does not change its capacity.

High Variance

Model is flexible but unstable. Data helps it “average out” and stabilize its shape.