Machine Learning
│
├── Foundational Theory
│ ├── Statistical Learning Theory — formalizes how models generalize from finite data to unseen examples
│ │ ├── Bias-Variance Tradeoff — balancing model complexity vs. sensitivity to training noise
│ │ ├── PAC Learning — defines when a concept is "probably approximately correctly" learnable
│ │ ├── VC Dimension — measures the capacity/complexity of a hypothesis class
│ │ └── Empirical Risk Minimization — learning as minimization of average loss on training data
│ ├── Information Theory — underpins uncertainty, compression, and feature relevance in ML
│ │ ├── Entropy — measures uncertainty or information content in a distribution
│ │ ├── KL Divergence — quantifies how one probability distribution differs from another
│ │ └── Mutual Information — captures statistical dependence between two variables
│ ├── Optimization Theory — the mathematical engine that trains every ML model
│ │ ├── Convex Optimization — guarantees global minimum; basis for SVMs and logistic regression
│ │ ├── Gradient Descent — iteratively moves parameters in the direction of steepest loss decrease
│ │ │ ├── Stochastic Gradient Descent (SGD) — uses single/mini-batch samples for faster updates
│ │ │ ├── Momentum — accumulates velocity in gradient direction to dampen oscillations
│ │ │ ├── Adam — adaptive learning rates per parameter; dominant optimizer in deep learning
│ │ │ └── Learning Rate Scheduling — varies step size over training to improve convergence
│ │ └── Second-Order Methods — use curvature (Hessian) for faster but costlier convergence
│ └── Probability & Bayesian Inference — treats model parameters and predictions as distributions
│ ├── Bayes Theorem — updates beliefs about parameters given observed data
│ ├── Maximum Likelihood Estimation — finds parameters maximizing the probability of observed data
│ ├── Maximum A Posteriori — MLE regularized by a prior distribution over parameters
│ └── Bayesian Networks — directed graphical models encoding conditional dependencies
│
├── Learning Paradigms
│ ├── Supervised Learning — learns a mapping from labeled input-output pairs
│ │ ├── Classification — predicts a discrete class label for each input
│ │ │ ├── Binary Classification — output is one of two classes (e.g., spam / not spam)
│ │ │ └── Multi-class / Multi-label — output is one of many, or several simultaneous labels
│ │ └── Regression — predicts a continuous numerical value for each input
│ │ ├── Linear Regression — models output as a weighted sum of input features
│ │ └── Non-linear Regression — captures curved relationships via kernels or neural networks
│ ├── Unsupervised Learning — discovers structure in data without explicit labels
│ │ ├── Clustering — groups similar data points together
│ │ │ ├── K-Means — partitions data into K clusters by centroid proximity
│ │ │ ├── DBSCAN — density-based clustering; handles noise and arbitrary shapes
│ │ │ └── Hierarchical Clustering — builds nested cluster tree (dendrogram)
│ │ ├── Dimensionality Reduction — compresses data to fewer dimensions preserving structure
│ │ │ ├── PCA — finds orthogonal axes of maximum variance
│ │ │ ├── t-SNE — non-linear projection optimized for 2D/3D visualization
│ │ │ ├── UMAP — faster non-linear reduction preserving global and local structure
│ │ │ └── Autoencoders — neural encoder-decoder that learns a compact latent representation
│ │ └── Density Estimation — models the underlying probability distribution of data
│ │ ├── Gaussian Mixture Models — mixture of Gaussians fit via EM algorithm
│ │ └── Kernel Density Estimation — non-parametric smoothed density from data points
│ ├── Semi-Supervised Learning — leverages small labeled + large unlabeled data together
│ │ ├── Self-Training — model iteratively labels its own high-confidence unlabeled examples
│ │ ├── Label Propagation — spreads labels through a graph based on data similarity
│ │ └── Consistency Regularization — enforces similar predictions under data perturbations
│ ├── Self-Supervised Learning — generates supervision signal from data structure itself
│ │ ├── Contrastive Learning — pulls similar views together, pushes dissimilar apart in embedding space
│ │ │ ├── SimCLR — contrastive framework using data augmentations and projection head
│ │ │ └── MoCo — momentum contrast with a memory bank of negative examples
│ │ ├── Masked Prediction — predicts masked portions of input (tokens, patches, spectrogram)
│ │ │ ├── BERT-style MLM — randomly masks tokens and predicts them from context
│ │ │ └── MAE — masks large image patches and reconstructs pixels with a ViT
│ │ └── Next-Token / Causal Prediction — predicts the next element in a sequence
│ ├── Reinforcement Learning — agent learns policy by interacting with an environment for reward
│ │ ├── Model-Free RL — learns directly from experience without an internal world model
│ │ │ ├── Q-Learning — tabular value-based method using Bellman equation updates
│ │ │ ├── Deep Q-Network (DQN) — approximates Q-function with a neural network
│ │ │ ├── Policy Gradient (REINFORCE) — directly optimizes parameterized policy via rewards
│ │ │ ├── Actor-Critic (A3C/A2C) — combines value baseline with policy gradient updates
│ │ │ └── Proximal Policy Optimization (PPO) — stable policy update with clipped objective
│ │ ├── Model-Based RL — learns a world model to plan or generate synthetic rollouts
│ │ │ ├── Dyna-Q — mixes real and model-simulated experience for sample efficiency
│ │ │ └── MuZero — learns dynamics, policy, and value jointly without explicit rules
│ │ └── Multi-Agent RL — multiple agents co-learn in a shared environment
│ └── Meta-Learning — learns to learn; adapts quickly to new tasks with few examples
│ ├── MAML — finds initialization that fine-tunes rapidly to new tasks
│ └── Prototypical Networks — classifies by distance to per-class prototype embeddings
│
├── Core Algorithms & Models
│ ├── Linear Models — simple, interpretable baselines with strong theoretical backing
│ │ ├── Logistic Regression — linear classifier trained via log-loss; outputs probabilities
│ │ ├── Ridge / Lasso Regression — L2 / L1 regularized linear regression for stable fits
│ │ └── Support Vector Machines (SVM) — max-margin classifier; kernel trick for non-linearity
│ ├── Tree-Based Models — partition feature space with axis-aligned splits
│ │ ├── Decision Tree — recursive feature splits; interpretable but prone to overfitting
│ │ ├── Random Forest — bagged ensemble of decorrelated trees for variance reduction
│ │ ├── Gradient Boosted Trees (GBT) — additive ensemble correcting residuals sequentially
│ │ │ ├── XGBoost — efficient GBT with second-order gradients and regularization
│ │ │ ├── LightGBM — leaf-wise growth and histogram binning for speed on large data
│ │ │ └── CatBoost — ordered boosting; native support for categorical features
│ │ └── Isolation Forest — anomaly detection by isolating points with random splits
│ ├── Probabilistic Models — model uncertainty explicitly; interpretable generative process
│ │ ├── Naive Bayes — fast classifier assuming feature independence given class
│ │ ├── Hidden Markov Models — sequential latent-state model for time-series / speech
│ │ └── Gaussian Processes — non-parametric Bayesian regression with uncertainty estimates
│ ├── Instance-Based Models — defer generalization; predict from stored training examples
│ │ ├── k-Nearest Neighbors (kNN) — classifies by majority vote of K closest examples
│ │ └── Locally Weighted Regression — fits local linear model weighted by query proximity
│ └── Neural Networks — hierarchical, differentiable function approximators trained end-to-end
│ ├── Feedforward Neural Network (MLP) — fully connected layers; universal approximator
│ ├── Convolutional Neural Network (CNN) — local filters exploit spatial/temporal structure
│ │ ├── ResNet — skip connections solve vanishing gradient in very deep networks
│ │ ├── EfficientNet — compound scaling of depth, width, resolution for efficiency
│ │ └── U-Net — encoder-decoder with skip connections for dense pixel prediction
│ ├── Recurrent Neural Network (RNN) — processes sequences with hidden state memory
│ │ ├── LSTM — gated cells mitigate vanishing gradient for long-range dependencies
│ │ └── GRU — simplified gating of LSTM with fewer parameters
│ ├── Transformer — self-attention over full sequence; foundation of modern AI
│ │ ├── Encoder-only (BERT) — deep bidirectional context for NLP understanding tasks
│ │ ├── Decoder-only (GPT) — autoregressive generation; backbone of large language models
│ │ ├── Encoder-Decoder (T5, BART) — sequence-to-sequence tasks like translation/summarization
│ │ └── Vision Transformer (ViT) — applies transformer directly to image patches
│ └── Graph Neural Network (GNN) — learns on graph-structured data via message passing
│ ├── GCN — aggregates neighbor features via normalized graph convolution
│ └── GAT — attention-weighted neighbor aggregation for adaptive importance
│
├── Generative Models — learn to synthesize new data samples from learned distributions
│ ├── Variational Autoencoder (VAE) — encodes data to latent Gaussian; decodes to reconstruct
│ ├── Generative Adversarial Network (GAN) — generator vs. discriminator adversarial game
│ │ ├── StyleGAN — style-based generator for high-fidelity controllable image synthesis
│ │ └── CycleGAN — unpaired image-to-image translation via cycle consistency loss
│ ├── Diffusion Models — iteratively denoise from Gaussian noise to data; state-of-the-art quality
│ │ ├── DDPM — denoising diffusion probabilistic model with fixed Markov chain
│ │ ├── Score Matching / SDE — continuous-time formulation via stochastic differential equations
│ │ └── Latent Diffusion (Stable Diffusion) — diffusion in compressed VAE latent space
│ └── Flow-Based Models — exact likelihood via invertible transformations (normalizing flows)
│
├── Model Development Process
│ ├── Data Engineering — acquiring, cleaning, and structuring data for learning
│ │ ├── Data Collection — web scraping, APIs, sensors, user logs, labeling pipelines
│ │ ├── Data Cleaning — handling missing values, outliers, duplicates, schema errors
│ │ ├── Feature Engineering — crafting informative input representations from raw data
│ │ │ ├── Encoding Categorical Variables — one-hot, ordinal, target, embedding encodings
│ │ │ ├── Feature Scaling — normalization / standardization for gradient-sensitive models
│ │ │ └── Feature Selection — filter, wrapper, and embedded methods to remove noise
│ │ └── Data Augmentation — artificially expanding dataset via transforms to reduce overfitting
│ ├── Model Selection & Evaluation — choosing and measuring the right model for the task
│ │ ├── Train / Validation / Test Split — partition data to estimate generalization performance
│ │ ├── Cross-Validation — k-fold averaging for robust performance estimates on small datasets
│ │ ├── Evaluation Metrics — task-specific scores to quantify model quality
│ │ │ ├── Classification Metrics — accuracy, precision, recall, F1, AUC-ROC, log-loss
│ │ │ ├── Regression Metrics — MAE, RMSE, R², MAPE
│ │ │ └── Ranking / Generation Metrics — NDCG, MAP, BLEU, ROUGE, FID
│ │ └── Hyperparameter Optimization — search over model configuration for best validation score
│ │ ├── Grid Search — exhaustive search over predefined hyperparameter grid
│ │ ├── Random Search — randomly sampled configurations; often more efficient than grid
│ │ ├── Bayesian Optimization — probabilistic surrogate model guides next evaluation
│ │ └── Neural Architecture Search (NAS) — automated discovery of network topology
│ ├── Regularization — prevents overfitting by constraining model complexity
│ │ ├── L1 / L2 Weight Decay — penalizes large weights; promotes sparsity (L1) or small norms (L2)
│ │ ├── Dropout — randomly zeros activations during training to prevent co-adaptation
│ │ ├── Early Stopping — halts training when validation loss stops improving
│ │ ├── Batch Normalization — normalizes layer activations for stable, faster training
│ │ └── Data Augmentation — (also serves as implicit regularizer; see Data Engineering)
│ └── Ensemble Methods — combines multiple models to reduce variance and improve accuracy
│ ├── Bagging — trains models on bootstrap samples; reduces variance (e.g., Random Forest)
│ ├── Boosting — sequentially correct errors of prior weak learners (e.g., XGBoost)
│ └── Stacking — meta-learner trained on base-model predictions as input features
│
├── Advanced Topics
│ ├── Transfer Learning — reuses knowledge from a source task/domain on a new task
│ │ ├── Fine-Tuning — updates pre-trained model weights on task-specific labeled data
│ │ ├── Feature Extraction — freezes backbone; trains only a new classification head
│ │ └── Domain Adaptation — bridges distribution shift between source and target domains
│ ├── Large Language Models (LLMs) — transformer decoders at scale; emergent general capabilities
│ │ ├── Pre-training — next-token prediction on massive corpora builds world knowledge
│ │ ├── Instruction Tuning — supervised fine-tuning on (instruction, response) pairs
│ │ ├── RLHF — human preference feedback shapes model to be helpful, harmless, honest
│ │ ├── Prompt Engineering — crafting inputs to elicit desired behavior without weight updates
│ │ │ ├── Few-Shot Prompting — examples in context guide output format and style
│ │ │ └── Chain-of-Thought — step-by-step reasoning traces improve complex problem solving
│ │ ├── Retrieval-Augmented Generation (RAG) — augments LLM with retrieved external knowledge
│ │ └── Parameter-Efficient Fine-Tuning (PEFT) — adapts LLMs cheaply without full fine-tune
│ │ ├── LoRA — low-rank weight delta matrices injected into attention layers
│ │ └── Prefix / Prompt Tuning — trains only soft continuous prompt tokens prepended to input
│ ├── Multimodal Learning — models that jointly process multiple data modalities
│ │ ├── CLIP — contrastive image-text alignment; zero-shot visual classification
│ │ ├── Flamingo / LLaVA — LLMs extended with visual encoder for image-grounded dialogue
│ │ └── Whisper — speech encoder + text decoder for robust multilingual transcription
│ ├── Federated Learning — trains across decentralized devices without sharing raw data
│ │ ├── FedAvg — aggregates local model updates by weighted averaging on central server
│ │ └── Differential Privacy — adds calibrated noise to gradients to protect individual records
│ └── Responsible AI — ensures models are safe, fair, and interpretable in practice
│ ├── Explainability & Interpretability — understanding why a model made a decision
│ │ ├── SHAP — game-theoretic feature attribution consistent across models
│ │ ├── LIME — locally fits interpretable surrogate around a single prediction
│ │ └── Attention Visualization — inspects transformer attention weights as saliency proxy
│ ├── Fairness & Bias — detects and mitigates discriminatory patterns in models
│ │ ├── Demographic Parity — equalize positive prediction rates across protected groups
│ │ └── Counterfactual Fairness — prediction unchanged if sensitive attributes were different
│ ├── Robustness & Adversarial ML — handles distribution shift and deliberate attacks
│ │ ├── Adversarial Training — includes adversarial examples in training to harden model
│ │ └── Out-of-Distribution Detection — flags inputs outside the training distribution
│ └── Model Compression — reduces model size and latency for edge / production deployment
│ ├── Pruning — removes low-importance weights or entire neurons/heads
│ ├── Quantization — represents weights/activations in lower-bit precision (INT8, FP16)
│ └── Knowledge Distillation — trains small student to mimic large teachers outputs
│
├── Tooling & Infrastructure
│ ├── Frameworks & Libraries
│ │ ├── PyTorch — dynamic-graph deep learning; dominant in research and production
│ │ ├── TensorFlow / Keras — static + eager execution; strong production ecosystem
│ │ ├── JAX — XLA-compiled NumPy with autograd; popular for LLM and RL research
│ │ ├── scikit-learn — clean API for classical ML algorithms; standard for tabular data
│ │ └── Hugging Face Transformers — model hub + training utilities for NLP/vision/audio
│ ├── Data & Feature Pipelines
│ │ ├── Pandas / Polars — dataframe manipulation; Polars is faster for large datasets
│ │ ├── DuckDB — in-process SQL analytics; fast for medium-scale ML data prep
│ │ ├── Apache Spark — distributed data processing for petabyte-scale feature engineering
│ │ └── Feature Stores (Feast, Tecton) — centralize feature computation and reuse across models
│ ├── Experiment Tracking & Versioning
│ │ ├── MLflow — open-source run tracking, model registry, and deployment abstraction
│ │ ├── Weights & Biases (W&B) — rich dashboards for metrics, artifacts, and hyperparameter sweeps
│ │ └── DVC — Git-based data and model version control for reproducible pipelines
│ └── MLOps & Serving
│ ├── Model Registries — version-controlled storage of trained model artifacts
│ ├── Serving Runtimes — infrastructure to serve predictions at scale
│ │ ├── TorchServe / TF Serving — framework-native production inference servers
│ │ ├── Triton Inference Server — multi-framework GPU serving with dynamic batching
│ │ └── vLLM — high-throughput LLM serving via PagedAttention memory management
│ ├── Orchestration (Kubeflow, Airflow) — schedules and monitors end-to-end ML pipelines
│ ├── Continuous Training — automatically retrains models when data or performance drifts
│ └── Monitoring & Observability — tracks prediction drift, data skew, and latency in production
│
└── Application Domains
├── Computer Vision — models that interpret and generate visual content
│ ├── Image Classification — assigns a label to a whole image (e.g., ResNet, ViT)
│ ├── Object Detection — localizes and classifies multiple objects (e.g., YOLO, DETR)
│ ├── Semantic / Instance Segmentation — pixel-level scene understanding (e.g., SAM)
│ └── Image / Video Generation — synthesizes realistic visuals (e.g., Stable Diffusion, Sora)
├── Natural Language Processing — models that understand and generate human language
│ ├── Text Classification — sentiment analysis, intent detection, topic labeling
│ ├── Named Entity Recognition — identifies people, places, and organizations in text
│ ├── Machine Translation — maps text from one language to another (e.g., NLLB, DeepL)
│ ├── Question Answering — retrieves or generates answers grounded in a context passage
│ ├── Text Summarization — condenses long documents into shorter key-point summaries
│ └── Code Generation — generates, completes, or explains code (e.g., Codex, Copilot)
├── Speech & Audio — models processing acoustic signals
│ ├── Automatic Speech Recognition (ASR) — converts spoken audio to text (e.g., Whisper)
│ ├── Text-to-Speech (TTS) — synthesizes natural-sounding speech from text
│ └── Speaker Diarization — segments audio by identifying "who spoke when"
├── Recommender Systems — predicts user preferences to surface relevant items
│ ├── Collaborative Filtering — leverages user-item interaction patterns (matrix factorization)
│ ├── Content-Based Filtering — matches item features to user profile attributes
│ └── Two-Tower / Retrieval-Ranking — scalable retrieve-then-rerank pipeline for large catalogs
├── Time-Series & Forecasting — models for temporal sequence prediction and anomaly detection
│ ├── Statistical Models (ARIMA, ETS) — classical decomposition of trend, seasonality, noise
│ ├── Temporal Fusion Transformer — attention-based multivariate forecasting with interpretability
│ └── Anomaly Detection — flags unusual patterns in streams (e.g., Isolation Forest, LSTM-AE)
└── Healthcare & Science — specialized high-stakes ML applications
├── Medical Imaging — diagnoses from X-ray, MRI, pathology slides via CNNs/ViTs
├── Protein Structure Prediction — AlphaFold2 predicts 3D protein fold from sequence
├── Drug Discovery — generative models propose novel molecular structures with target affinity
└── Clinical NLP — extracts structured data from EHR notes; FHIR integration for interoperability
Tooling & Infrastructure
│
├── Frameworks & Libraries
│ ├── Deep Learning Frameworks
│ │ ├── PyTorch — dynamic autograd graph; dominant in both research and production DL
│ │ │ ├── torch.nn — module system for defining layers and custom architectures
│ │ │ ├── torch.autograd — automatic differentiation engine for gradient computation
│ │ │ ├── torch.compile — JIT compilation via TorchDynamo/Inductor for faster execution
│ │ │ ├── TorchScript — serializes PyTorch models to a static graph for deployment
│ │ │ ├── torch.distributed — primitives for multi-GPU and multi-node training
│ │ │ └── PyTorch Lightning — high-level training loop wrapper reducing boilerplate code
│ │ ├── TensorFlow / Keras — static + eager execution graph; strong production ecosystem
│ │ │ ├── tf.keras — high-level API for rapid model prototyping and training
│ │ │ ├── tf.data — efficient input pipeline for loading and preprocessing datasets
│ │ │ ├── TFX (TensorFlow Extended) — end-to-end production ML pipeline framework
│ │ │ └── TensorFlow Lite — converts models for mobile and edge device inference
│ │ └── JAX — XLA-compiled NumPy + autograd; favored for LLM and RL research
│ │ ├── Flax — neural network library built on JAX with functional design
│ │ ├── Optax — gradient processing and optimizer library for JAX
│ │ └── vmap / pmap / jit — JAX primitives for vectorization, parallelism, and JIT
│ ├── Classical ML Libraries
│ │ ├── scikit-learn — clean API for classical algorithms; standard for tabular data
│ │ ├── XGBoost — gradient boosted trees with L1/L2 regularization; tabular SOTA
│ │ ├── LightGBM — histogram-based GBT; fast and memory-efficient on large datasets
│ │ ├── CatBoost — ordered boosting with native categorical feature support
│ │ └── statsmodels — statistical modeling and hypothesis testing for ML practitioners
│ ├── Scientific Computing Stack
│ │ ├── NumPy — foundational N-dimensional array library; backbone of Python ML
│ │ ├── SciPy — optimization, integration, linear algebra, and signal processing routines
│ │ ├── Einops — expressive tensor dimension manipulation used in modern DL code
│ │ └── Numba — JIT compiler for NumPy-heavy Python loops via LLVM
│ └── Specialized DL Libraries
│ ├── Hugging Face Transformers — model hub + training utilities for NLP/vision/audio
│ ├── timm (PyTorch Image Models) — large collection of pretrained vision models for CV
│ ├── torchvision / torchaudio — domain-specific datasets, transforms, and model zoos
│ ├── Detectron2 — Metas research platform for object detection and segmentation
│ ├── OpenMMLab (MMDetection, MMSeg) — modular CV research framework by Shanghai AI Lab
│ └── Sentence-Transformers — easy dense embedding extraction for similarity and retrieval
│
├── Data & Feature Pipelines
│ ├── Data Loading & Preprocessing
│ │ ├── Pandas — dataframe library; standard for exploratory data analysis and feature prep
│ │ ├── Polars — Rust-backed dataframe; significantly faster than Pandas on large tables
│ │ ├── DuckDB — in-process SQL analytics; ideal for medium-scale ML data preparation
│ │ ├── Apache Arrow — columnar in-memory format enabling zero-copy data interchange
│ │ └── NVIDIA RAPIDS (cuDF) — GPU-accelerated dataframe processing, drop-in Pandas API
│ ├── Large-Scale Processing
│ │ ├── Apache Spark — distributed compute for petabyte-scale feature engineering
│ │ ├── Apache Flink — stream-first distributed processing for real-time feature pipelines
│ │ └── Ray Data — distributed data loading and preprocessing tightly integrated with Ray
│ ├── Dataset Management
│ │ ├── Hugging Face Datasets — versioned datasets with Arrow-backed fast loading
│ │ ├── TensorFlow Datasets (TFDS) — standardized dataset loading for TF/JAX workflows
│ │ ├── FFCV — fast dataset format using memory-mapped files for CPU-bottleneck training
│ │ └── WebDataset — tar-based streaming dataset format for large-scale training on cloud
│ ├── Feature Stores
│ │ ├── Feast — open-source feature store for offline + online feature serving
│ │ ├── Tecton — managed feature platform with time-travel and point-in-time joins
│ │ └── Hopsworks — unified feature store with built-in training pipeline integration
│ └── Data Annotation & Labeling
│ ├── Label Studio — open-source multi-type annotation tool (image, text, audio, video)
│ ├── Labelbox — enterprise platform with AI-assisted labeling and workforce management
│ ├── Scale AI — API-based human-in-the-loop labeling service for large ML datasets
│ └── V7 Darwin — annotation platform with auto-labeling and dataset versioning
│
├── Training Infrastructure
│ ├── Distributed Training
│ │ ├── Data Parallelism — splits batches across GPUs; each holds full model replica
│ │ │ ├── torch.nn.DataParallel — simple single-node multi-GPU parallelism in PyTorch
│ │ │ └── DistributedDataParallel (DDP) — efficient multi-node gradient sync via NCCL
│ │ ├── Model Parallelism — splits model layers across devices when model exceeds GPU VRAM
│ │ │ ├── Pipeline Parallelism — stages of a model placed on sequential GPU ranks
│ │ │ └── Tensor Parallelism — individual layer weight matrices sharded across GPUs
│ │ ├── DeepSpeed — Microsoft library for ZeRO optimizer, offloading, and mixed precision
│ │ │ ├── ZeRO Stage 1/2/3 — shards optimizer states, gradients, and parameters across GPUs
│ │ │ └── Offloading (CPU/NVMe) — moves optimizer states to CPU RAM to fit larger models
│ │ ├── FSDP (Fully Sharded Data Parallel) — PyTorch-native ZeRO-3 equivalent for large models
│ │ └── Megatron-LM — NVIDIAs 3D-parallel framework for training trillion-parameter models
│ ├── Mixed Precision & Memory Efficiency
│ │ ├── AMP (Automatic Mixed Precision) — trains in FP16/BF16 with FP32 master weights
│ │ ├── BF16 Training — brain float format; better numeric range than FP16 for stability
│ │ ├── Gradient Checkpointing — recomputes activations on backward pass to save VRAM
│ │ └── Flash Attention — IO-aware exact attention implementation; dramatically cuts memory use
│ ├── Compute Hardware
│ │ ├── NVIDIA GPUs — CUDA-accelerated; A100 / H100 / H200 / B200 are current training targets
│ │ │ ├── CUDA — low-level parallel computing platform and API for NVIDIA GPU programming
│ │ │ ├── cuDNN — GPU-accelerated primitives for DNNs (convolutions, RNNs, attention)
│ │ │ └── NCCL — NVIDIA collective communication library for multi-GPU gradient allreduce
│ │ ├── Google TPUs — custom matrix multiply units; dominant in large-scale JAX/TF training
│ │ │ └── XLA Compiler — domain-specific compiler optimizing tensor computations for TPU/GPU
│ │ ├── AMD ROCm — open GPU compute platform; ROCm enables CUDA-like ML on AMD GPUs
│ │ └── Apple Silicon (MPS backend) — Metal Performance Shaders backend for Mac-local training
│ └── Hyperparameter & Experiment Orchestration
│ ├── Ray Tune — scalable hyperparameter search with many HPO algorithms built in
│ ├── Optuna — Bayesian / CMA-ES HPO with pruning of unpromising trials mid-training
│ └── Hydra — hierarchical config system for managing complex training configurations
│
├── Experiment Tracking & Reproducibility
│ ├── MLflow — open-source run tracking, model registry, and deployment abstraction layer
│ ├── Weights & Biases (W&B) — rich dashboards, artifact versioning, and sweep-based HPO
│ ├── Neptune.ai — metadata store optimized for long experiment histories and team logging
│ ├── Comet ML — experiment tracking with built-in model evaluation and data panels
│ ├── DVC (Data Version Control) — Git-based versioning of data, models, and pipeline stages
│ └── Reproducibility Tooling
│ ├── Docker / NVIDIA Docker — containerizes training environment for exact reproduction
│ ├── Conda / pip-tools — pins Python dependency graphs to lock environment exactly
│ └── Deterministic Seeds — controlling RNG seeds in torch/numpy/CUDA for reproducibility
│
├── Model Serving & Inference Runtimes
│ ├── Framework-Native Servers
│ │ ├── TorchServe — PyTorchs official model server with handler API and REST endpoints
│ │ ├── TF Serving — TensorFlows gRPC + REST server with versioned model management
│ │ └── BentoML — framework-agnostic model packaging, API wrapping, and container export
│ ├── High-Performance Inference Engines
│ │ ├── ONNX Runtime — cross-framework inference; converts models to ONNX for portability
│ │ │ └── ONNX — open standard for representing ML model graphs across frameworks
│ │ ├── TensorRT — NVIDIAs inference optimizer; layer fusion, quantization, FP8 for GPUs
│ │ ├── TensorRT-LLM — TensorRT-based runtime for LLMs; in-flight batching, paged KV cache
│ │ ├── vLLM — high-throughput LLM serving via PagedAttention; best for many concurrent users
│ │ ├── SGLang — structured generation runtime; RadixAttention for low per-token latency
│ │ ├── Hugging Face TGI — open-source LLM serving with continuous batching and quantization
│ │ ├── llama.cpp — C++ inference runtime for transformer models on CPU and consumer GPUs
│ │ └── OpenVINO — Intels inference toolkit optimized for CPU/NPU/iGPU edge deployment
│ ├── Multi-Framework Serving
│ │ └── Triton Inference Server — NVIDIAs multi-framework GPU server with dynamic batching
│ └── Edge & On-Device Inference
│ ├── TensorFlow Lite — optimized runtime for mobile and embedded devices
│ ├── CoreML — Apples on-device inference runtime for iOS / macOS applications
│ ├── ONNX Runtime Mobile — cross-platform edge inference for Android / iOS
│ └── ExecuTorch — PyTorchs official on-device inference stack for edge / mobile
│
├── Model Optimization
│ ├── Quantization
│ │ ├── Post-Training Quantization (PTQ) — quantizes a trained model without retraining
│ │ │ ├── GPTQ — one-shot weight quantization for transformers using second-order info
│ │ │ └── AWQ — activation-aware weight quantization preserving salient weight channels
│ │ └── Quantization-Aware Training (QAT) — simulates low-precision in forward pass during training
│ ├── Pruning
│ │ ├── Unstructured Pruning — zeros out individual weights based on magnitude or gradient
│ │ └── Structured Pruning — removes entire channels, heads, or layers for hardware speedup
│ ├── Knowledge Distillation
│ │ ├── Response Distillation — student matches teachers soft output logits
│ │ └── Feature Distillation — student matches intermediate teacher layer representations
│ ├── Neural Architecture Search (NAS)
│ │ ├── DARTS — differentiable NAS; architecture search via gradient descent
│ │ └── EfficientNet-style Compound Scaling — systematic scaling of depth, width, resolution
│ └── Compilation & Graph Optimization
│ ├── torch.compile (TorchInductor) — fuses ops and generates optimized GPU kernels
│ ├── XLA — accelerated linear algebra compiler used by TensorFlow and JAX
│ └── Apache TVM — deep learning compiler stack targeting diverse hardware backends
│
├── MLOps & Pipeline Orchestration
│ ├── Pipeline Orchestration
│ │ ├── Kubeflow Pipelines — Kubernetes-native ML workflow orchestration with DAG UI
│ │ ├── Apache Airflow — general-purpose DAG scheduler; widely used for ETL and training jobs
│ │ ├── Prefect — modern Python-native workflow orchestration with dynamic task graphs
│ │ ├── Metaflow — Netflix-originated framework; local + cloud execution with versioned steps
│ │ └── ZenML — ML pipeline abstraction that is stack-agnostic and cloud-portable
│ ├── Model Registry & Artifact Storage
│ │ ├── MLflow Model Registry — stages models through dev / staging / production lifecycle
│ │ ├── W&B Artifacts — versioned storage for datasets, models, and evaluation results
│ │ └── Hugging Face Hub — public + private model and dataset registry with Git-LFS backend
│ ├── CI/CD for ML
│ │ ├── DVC Pipelines — defines reproducible, cacheable stage-based training pipelines
│ │ ├── CML (Continuous Machine Learning) — GitHub Actions extension for ML CI/CD reporting
│ │ └── Vertex AI Pipelines / SageMaker Pipelines — managed cloud-native ML CI/CD services
│ ├── Infrastructure Provisioning
│ │ ├── Kubernetes — container orchestration; standard substrate for production ML systems
│ │ ├── Ray — distributed Python runtime for training, tuning, serving, and data processing
│ │ └── Slurm — HPC workload manager for on-prem GPU cluster job scheduling
│ └── Monitoring & Observability
│ ├── Evidently AI — open-source ML monitoring for data drift and model performance
│ ├── Fiddler AI — production model monitoring with explainability and bias detection
│ ├── WhyLabs — continuous data and model monitoring with statistical drift profiles
│ └── Prometheus + Grafana — infra-level metrics collection and dashboard visualization
