Basic Maths — logs, exponents, functions, bias, geometry, and trig. Recall what a function and a bias are before any model. The calculus heading at the bottom of this note is the next file.
Algebra Vectors & matrics — vectors, dot product, span, basis, rank, null space, determinants, eigenvectors. Recall a matrix as a linear transformation. Solving a system is the next note.
System of linear equation — Gaussian elimination and singular systems. Recall rank and determinant as the test for “one solution”.
Calculs — slope, limits, derivative rules, chain rule, partial derivatives. Recall the chain rule; it is backprop. The perceptron section at the end is the bridge into a loss.
Probability — axioms, conditional probability, Bayes, random variables, distributions. Recall Bayes and the difference between a mass function and a density.
Statistics — descriptive versus inferential, outliers, five-number summary. Recall mean, variance, and why an outlier matters before you normalize data.
Markov Chain — transition probabilities, stationary distribution, then a tiny language-model example. Recall the Markov property: the next step depends on the current state.
Information theory — outline of entropy, cross-entropy, KL, mutual information, plus the paper link. Missing: the prose for those terms. The KL walk-through that does exist is inside Generative Modeling.
Information theory — pointer. The paper link lives on the mathematics note.
2. Machine learning concepts
Machine learning workflow — data, features, model, loss, optimization, regularization, training, evaluation, inference. Recall which box you are in before opening an algorithm note.
ML Roadmap — map of theory, paradigms, and applications. It is an outline.
Supervised Learning — linear and polynomial regression, logistic regression, regularization (ridge and lasso), data augmentation, SVM, Decision Trees, entropy and information gain, random forests, naive Bayes, ensembles. Recall logistic regression as a linear model plus a sigmoid, and entropy as the split criterion in a tree.
Loss function in ML — regression losses and classification losses, including cross-entropy. Recall the loss as the number training is trying to push down.
Optimization and Hyperparameter in ML — gradient as the local slope, SGD, Adam, learning rate, momentum, batch size. Recall why the update subtracts the gradient.
Softmax — softmax turns scores into probabilities. Read the rest of that note (vanishing gradients, initialization, gradient checking, mini-batch, exponentially weighted averages) when you reach training in the next stage.
Feature Engineering — filter, wrapper, and embedded selection, missing values, categorical features. Recall that the model only sees the columns you hand it. The short feature-engineering heading inside Machine learning points here.
Model Evaluation — outline of regression and classification metrics. Missing: a prose walk-through of MAE, MSE, precision, recall, and F1. Those names return in LLM Observability And Eval.
My Notes — one Kaggle / feature-engineering reminder.
AI insights — Karpathy on tasks that have a score versus tasks that do not.
3. Deep learning foundations
Deep Learning RoadMap — map from the neuron through training, architectures, and later topics. It is an outline. Where a branch has no note below, that branch is missing as prose.
Perceptron — one neuron, then the multi-input formula. Recall weighted sum, bias, activation.
Universal Approximation Theorem — enough neurons and a nonlinear activation can approximate a function. Recall why a stack of purely linear layers cannot.
Gradient — a one-weight network, a loss, and why training multiplies by the gradient. Recall the sign of the update.
Regularization — dropout, batch norm, layer norm, gradient clipping, Weight intilization, residual connections. Recall a residual link as a path that lets the gradient skip layers. The exponentially weighted average example points at Activation functions.
Pytroch — tensors, shapes, and a training step. Use this to redo the perceptron by hand in code.
Tensorflow — constants, variables, graphs, and gradients in TensorFlow.
Convolutional Neural Network — kernels, padding, stride, pooling, a backward pass, residual block. Recall a kernel as a small sliding dot product.
RNN (Recurrent Neural Network) — forward equations, BPTT, RNN shapes, next-word prediction. Recall the hidden state as the memory passed to the next step.
Natural language processing — morphology through pragmatics, parsing, n-grams, TF-IDF, word2vec. Recall TF-IDF as “common in this document, rare in the corpus”.
Text classifcation NLP — naive Bayes on TF-IDF, then a confusion matrix. Recall the matrix before you trust an accuracy number.
Transformer — read Position encoding, Encoder, Self -Attention, multi-head attention, the feed-forward block, and Decoder before the type list at the top. Recall attention as a weighted sum of values, with weights from query-key similarity. The same note also holds mixture-of-experts, state-space models, causal attention, logits, and BERT.
Whisper — one full encoder-decoder speech model: windows, encoder, decoder, cross-attention, tokens. Recall cross-attention as the decoder reading the encoder.
5. Generative models and reinforcement learning
Generative Modeling — discriminative versus generative, autoencoders, variational autoencoders, the reparameterization trick, reconstruction loss, and KL. This is the KL lesson the information-theory outline does not contain. Recall the bottleneck as a distribution, not a single point.
Reinforcement learning — the agent loop, return, discount, policy, and the moved lesson on state value, action value, and the Bellman equation. Recall return as discounted future reward, and the Bellman equation as “now plus discounted next”.
Reasoning in LLM — inference-time scaling, distillation, then the RL pointer, then the illusion-of-thinking notes and the Anthropic chain-of-thought faithfulness notes. Recall test-time compute as spending more search at answer time without changing weights.
6. Training and adapting language models
Neurons indepth — the Chinchilla scaling law: parameters and tokens move together. Recall about twenty tokens per parameter for the dense compute-optimal point.
PreTraining LLM — a stub. Missing: a pretraining lesson (objective, data, the training loop).
Fine-Tunning — read supervised finetuning, then LoRA and QLoRA, then preference alignment and DPO, then Model Merging (SLURP, DeRA, pass-through, mixture of experts). Tool notes (Axolotl, Unsloth, Lamini, Ray) sit in the same file after the methods.
Large Language Model — 1-bit weights (BitNet). Merging points at Fine-Tunning. The resource list at the bottom is a reading pile.
Model Quantization — integer grids, block size, GGUF names, dequantization. Recall quantization as storing weights in fewer bits and dequantizing them for the multiply.
Building reasoning model from Scratch — short list: self-consistency, chain of thought, self-refinement. The longer versions are in the prompting stage.
7. Retrieval and RAG
Information Retrieval — boolean search, bag of words, TF-IDF, BM25, inverted index, sentence transformers. Recall BM25 as the lexical ranker you compare embeddings against.
Vector DB and embedding — dense and sparse vectors, HNSW, IVF, LSH, then specific databases. Recall a vector index as a way to avoid comparing the query to every row.
Encoder and reranker — bi-encoder versus cross-encoder. Recall the bi-encoder as “encode once, compare vectors” and the cross-encoder as “score this pair together”.
RAG — the retrieve-then-generate loop, chunking, retrieval variants, and the later named systems (fusion, CRAG, LightRAG, HippoRAG, and the rest). Recall the failure mode: the answer is only as good as the retrieved passages. Graph RAG points at the next note.
Graph Rag — knowledge-graph definition, then entity extraction, community query, and updates. Recall nodes as entities and edges as relations.
Generative AI — unstructured documents into a pipeline, and long-text summarization.
Prompt Engineering — prompt patterns, then chain of thought, self-consistency, ReAct, tree of thoughts. Recall chain of thought as writing the intermediate steps in the context. DSPy and the product prompts later in the same note are the applied half.
LLM Hacking — prompt injection, jailbreaks, and why safety filters miss them. Recall that the model follows the text in the context, including text an attacker put there.
9. Inference and running models
How Model is partitioned — config, weights, layer map, tokenizer. Recall that the weights file and the tokenizer are separate artifacts.
GPU — streaming multiprocessors, memory for parameters, FLOPs, clusters.
LLM Inference — speculative decoding, KV cache, prefix cache, continuous batching, paged attention. Recall the KV cache as “do not recompute keys and values for tokens you already emitted”.
Model Paramerter — temperature, top-k, top-p. Recall temperature as sharpening or flattening the next-token distribution.
10. Evaluation
LLM Observability And Eval — perplexity, precision/recall/F1, SFT and preference evals, LLM-as-judge, agent-as-judge. Recall perplexity as how surprised the model is by the next token.
Eval on chat bot — task completion, coherence, and context retention for a chatbot.
11. Memory and context
Context Engineering and Memory in LLM — what goes in the window, long context, and compression. Recall the context window as the only state the model sees on that call.
Congnee — cognee’s add, cognify, search, and memify pipeline.
Memo — Mem0’s memory types and add/search pipeline.
12. Vision and model case studies
Image AI roadmap — outline from classical vision to generative image models. The convnet lesson is Convolutional Neural Network. Missing: prose for the classical vision branches that the roadmap names and the convnet note does not cover.
Computer vision — vision-language models: image patches, a connector, then the language model. Recall a patch as the image version of a token.
