Image & Vision AI
│
├── Classical Computer Vision
│ ├── Edge Detection — Detecting boundaries (Sobel, Canny operators)
│ ├── Feature Descriptors
│ │ ├── SIFT — Scale-Invariant Feature Transform for keypoint matching
│ │ ├── SURF — Speeded-Up Robust Features (faster SIFT alternative)
│ │ ├── HOG — Histogram of Oriented Gradients for object shape description
│ │ └── ORB — Oriented FAST and Rotated BRIEF for real-time feature matching
│ ├── Template Matching — Sliding a template across image to find matches
│ ├── Optical Flow — Estimating pixel motion between consecutive frames
│ └── Image Filtering — Convolution kernels for blur, sharpen, emboss operations
│
├── Image Classification
│ │ — Assigning a single label to an entire image
│ │
│ ├── CNN Architectures (Evolution)
│ │ ├── LeNet-5 (1998) — First practical CNN for digit recognition
│ │ ├── AlexNet (2012) — Deep CNN that won ImageNet; launched the deep learning era
│ │ ├── VGGNet (2014) — Very deep (16-19 layers) with uniform 3×3 convolutions
│ │ ├── GoogLeNet/Inception (2014) — Multi-scale convolutions via inception modules
│ │ ├── ResNet (2015) — Skip connections enabling 100+ layer training
│ │ ├── DenseNet (2017) — Dense connections: each layer receives all preceding features
│ │ ├── MobileNet — Depthwise separable convolutions for mobile/edge deployment
│ │ ├── EfficientNet (2019) — Compound scaling of depth, width, and resolution
│ │ └── ConvNeXt (2022) — Modernized pure-CNN competing with Transformers
│ │
│ └── Transformer-based Classification
│ ├── ViT (Vision Transformer) — Splits image into patches, processes as tokens;
│ │ proved Transformers work for vision at scale
│ ├── DeiT — Data-efficient ViT using knowledge distillation
│ ├── Swin Transformer — Hierarchical ViT with shifted windows for multi-scale
│ └── 🆕 BEiT — BERT-style pre-training for vision Transformers
│
├── Object Detection
│ │ — Locating and classifying multiple objects with bounding boxes
│ │
│ ├── Two-Stage Detectors
│ │ ├── R-CNN — Region proposals + CNN classification (slow, pioneering)
│ │ ├── Fast R-CNN — Shared CNN features with RoI pooling
│ │ ├── Faster R-CNN — Region Proposal Network (RPN) makes proposals learnable
│ │ └── 🆕 Cascade R-CNN — Multi-stage refinement with increasing IoU thresholds
│ │
│ ├── One-Stage Detectors
│ │ ├── YOLO (v1→v11) — "You Only Look Once"; single forward pass for detection
│ │ ├── SSD — Multi-scale feature maps for detecting objects of various sizes
│ │ └── RetinaNet — Focal loss addressing class imbalance in dense detection
│ │
│ └── Transformer-based Detection
│ ├── DETR — End-to-end detection with Transformers and bipartite matching loss
│ ├── 🆕 DINO (Detection) — DETR variant with denoising anchors for faster convergence
│ ├── 🆕 Grounding DINO — Open-set detector accepting text queries
│ │ to detect arbitrary objects described in language
│ └── 🆕 OWL-ViT — CLIP-based open-vocabulary detector
│
├── Image Segmentation
│ │ — Pixel-level classification of image regions
│ │
│ ├── Semantic Segmentation (every pixel gets a class label)
│ │ ├── FCN (Fully Convolutional Network) — First end-to-end pixel-wise prediction
│ │ ├── U-Net — Encoder-decoder with skip connections; dominant in medical imaging
│ │ ├── DeepLab (v1-v3+) — Atrous/dilated convolutions + CRF for precise boundaries
│ │ ├── PSPNet — Pyramid Pooling Module for multi-scale context aggregation
│ │ └── SegFormer — Transformer-based efficient segmentation
│ │
│ ├── Instance Segmentation (each object instance gets its own mask)
│ │ ├── Mask R-CNN — Faster R-CNN + parallel mask prediction branch with RoI Align
│ │ └── 🆕 SOLO / SOLOv2 — Instance segmentation without RoI operations
│ │
│ ├── Panoptic Segmentation (combines semantic + instance)
│ │ └── 🆕 Mask2Former — Unified architecture for all segmentation types
│ │
│ └── Foundation Segmentation Models
│ ├── SAM (Segment Anything, Meta) — Prompt-driven universal segmentation;
│ │ accepts points, boxes, or masks as prompts for zero-shot segmentation
│ ├── SAM 2 — Extended to video with temporal memory for consistent mask tracking
│ └── 🆕 SAM 3 — Concept-driven multimodal segmentation (text + visual prompts)
│
├── 🆕 Vision Foundation Models
│ │ — Pre-trained models serving as universal backbones
│ │
│ ├── Contrastive Models
│ │ ├── CLIP (OpenAI) — Contrastive Language-Image Pretraining; aligns vision and
│ │ │ text encoders so image of "dog" is close to text "a photo of a dog"
│ │ └── SigLIP — Sigmoid loss variant of CLIP for better scaling
│ │
│ ├── Self-Supervised Vision Models
│ │ ├── DINO — Self-distillation with no labels; learns rich visual features from
│ │ │ unlabeled images using teacher-student framework
│ │ ├── DINOv2 (Meta) — Stronger DINO; no fine-tuning needed; learns features
│ │ │ including depth estimation that CLIP cannot capture
│ │ ├── MAE (Masked Autoencoder) — Masks 75% of image patches, reconstructs them;
│ │ │ learns strong representations through reconstruction
│ │ └── 🆕 I-JEPA — Joint Embedding Predictive Architecture predicting abstract
│ │ representations rather than pixels
│ │
│ └── Multimodal Embedding Models
│ ├── ImageBind (Meta) — Binds 6 modalities (image, text, audio, depth, thermal, IMU)
│ │ into a single shared embedding space
│ └── CLAP — Contrastive Language-Audio Pretraining; aligns audio with text
│ in shared space for zero-shot audio classification
│
├── Image Generation
│ │ — Creating new images from text, noise, or other images
│ │
│ ├── GAN-based Generation
│ │ ├── GAN — Generator vs. Discriminator adversarial training
│ │ ├── DCGAN — Deep Convolutional GAN with architectural guidelines
│ │ ├── StyleGAN / StyleGAN2 / StyleGAN3 — Style-based generator for
│ │ │ photorealistic face synthesis with disentangled control
│ │ ├── ProGAN — Progressive growing from low to high resolution
│ │ ├── CycleGAN — Unpaired image-to-image translation (e.g., horse→zebra)
│ │ ├── Pix2Pix — Paired image-to-image translation with conditional GAN
│ │ └── 🆕 GigaGAN — Scaling GANs to text-to-image at 1B+ parameters
│ │
│ ├── Diffusion-based Generation
│ │ ├── DDPM (Denoising Diffusion Probabilistic Model) — Forward process adds
│ │ │ Gaussian noise; reverse process learns to denoise step by step
│ │ ├── DDIM — Deterministic sampling variant of DDPM for faster inference
│ │ ├── Score-based Models (SDE) — Continuous-time diffusion via stochastic
│ │ │ differential equations
│ │ ├── Latent Diffusion (LDM) — Diffusion in compressed latent space (not pixel space)
│ │ │ for dramatic compute savings; foundation of Stable Diffusion
│ │ ├── Classifier Guidance — External classifier gradient steers generation
│ │ │ toward desired class
│ │ ├── Classifier-Free Guidance (CFG) — Jointly trains conditional + unconditional
│ │ │ model; mixes their scores for quality-diversity tradeoff
│ │ ├── 🆕 Noise Scheduling — Defines how noise is added/removed; critical for
│ │ │ sample quality (linear, cosine, etc.)
│ │ └── 🆕 Diffusion Transformer (DiT) — Replaces U-Net backbone with
│ │ Transformer for better scaling
│ │
│ ├── Notable Image Generation Models
│ │ ├── DALL-E / DALL-E 2 / DALL-E 3 (OpenAI) — Text-to-image; DALL-E 3 has
│ │ │ best text rendering in generated images
│ │ ├── Stable Diffusion (Stability AI) — Open-source latent diffusion; SDXL and
│ │ │ SD 3.5 with LoRA fine-tuning and ControlNet composability
│ │ ├── Midjourney — Proprietary; excels at artistic, polished aesthetic output
│ │ ├── Flux (Black Forest Labs) — Open-weight; excellent photorealism and
│ │ │ prompt adherence; Schnell variant for 1-4 step fast generation
│ │ ├── Imagen (Google) — Cascaded diffusion for high-fidelity text-to-image
│ │ └── 🆕 Ideogram — Specialized in accurate text rendering within images
│ │
│ └── 🆕 Image Generation Control Techniques
│ ├── ControlNet — Adds spatial conditioning (pose, depth, edges) to diffusion models
│ ├── LoRA (Low-Rank Adaptation) — Efficient fine-tuning with small trainable matrices
│ ├── IP-Adapter — Image Prompt Adapter for style/subject transfer
│ ├── Inpainting — Filling in masked/erased regions of an image
│ └── Outpainting — Extending an image beyond its original boundaries
│
├── 🆕 Vision-Language Models (VLMs)
│ │ — Models that understand images through natural language
│ │
│ ├── Architecture Patterns
│ │ ├── Vision Encoder + LLM — Frozen/fine-tuned vision encoder (CLIP/SigLIP)
│ │ │ projects features into LLM input space via projection layer
│ │ ├── Cross-Attention Fusion — Vision features attend to LLM hidden states
│ │ └── 🆕 Generative Vision Encoder — Florence-2 uses task-specific prompts
│ │ (caption, OCR, grounding) to extract multi-level features
│ │
│ ├── Notable VLMs
│ │ ├── LLaVA — Visual instruction tuning; CLIP ViT encoder + LLaMA LLM +
│ │ │ trainable projection matrix; 85.1% relative score vs GPT-4
│ │ ├── GPT-4V / GPT-4o — OpenAI's proprietary multimodal model; excels at
│ │ │ visual reasoning, OCR, and complex scene understanding
│ │ ├── BLIP-2 — Bridges frozen image encoder and LLM via lightweight Q-Former
│ │ ├── InternVL — Open-source VLM with strong benchmark performance
│ │ ├── Qwen-VL — Alibaba's vision-language model family
│ │ ├── Florence-VL — Uses Florence-2 generative vision encoder with
│ │ │ depth-breadth fusion for richer visual features than CLIP
│ │ ├── 🆕 Gemini (Google) — Natively multimodal; processes interleaved text,
│ │ │ image, audio, and video tokens
│ │ └── 🆕 Claude Vision (Anthropic) — Strong at chart/diagram comprehension
│ │
│ └── VLM Tasks
│ ├── Visual Question Answering (VQA) — Answering questions about images
│ ├── Image Captioning — Generating text descriptions of images
│ ├── Visual Grounding — Locating objects in image based on text description
│ └── Visual Reasoning — Multi-step logical inference about visual content
│
└── 🆕 Image Evaluation Metrics
├── FID (Fréchet Inception Distance) — Measures realism by comparing feature distributions
├── IS (Inception Score) — Measures quality and diversity of generated images
├── CLIP Score — Measures text-image alignment using CLIP embeddings
├── LPIPS — Learned Perceptual Image Patch Similarity
└── IoU / mAP — Intersection over Union / mean Average Precision for detection
OCR & Document AI
│
├── Optical Character Recognition (OCR)
│ │ — Extracting text from images
│ │
│ ├── Classical OCR
│ │ ├── Template Matching — Comparing character shapes against stored templates
│ │ ├── Feature Extraction (Rule-based) — HOG, SIFT features for character recognition
│ │ └── Statistical Methods (1990s) — Treating reading as inference; using
│ │ language context when uncertain about characters
│ │
│ ├── Deep Learning OCR
│ │ ├── CRNN — CNN for visual features + Bi-LSTM for sequence modeling + CTC for
│ │ │ alignment-free text decoding
│ │ ├── 🆕 STN (Spatial Transformer Network) — Learns to rectify/deskew text
│ │ │ before recognition
│ │ ├── 🆕 Attention-based OCR — Encoder-decoder with attention replacing CTC
│ │ │ for better handling of complex layouts
│ │ └── TrOCR (Microsoft) — Pure Transformer encoder-decoder for OCR;
│ │ pre-trained on printed and handwritten text
│ │
│ ├── Scene Text Detection (finding text in natural images)
│ │ ├── EAST — Efficient and Accurate Scene Text detector
│ │ ├── CRAFT — Character Region Awareness for Text detection
│ │ └── 🆕 DBNet — Differentiable Binarization for fast text detection
│ │
│ ├── Notable OCR Systems
│ │ ├── Tesseract — Open-source; HP→Google; LSTM-based engine (v4/5); supports
│ │ │ 100+ languages; uses CTC decoder with optional dictionary post-processing
│ │ ├── PaddleOCR — Baidu's open-source; strong for CJK languages
│ │ ├── EasyOCR — Simple Python library supporting 80+ languages
│ │ ├── Surya — Open-source OCR optimized for 90+ languages
│ │ └── 🆕 DocTR — Document Text Recognition; end-to-end detection + recognition
│ │
│ └── 🆕 LLM-based OCR
│ ├── GPT-4o OCR — Multimodal LLM directly reading documents with high accuracy
│ ├── Gemini OCR — Outperforms traditional OCR on historical and complex documents
│ └── Qwen-VL OCR — Strong OCR capability integrated into vision-language model
│
├── 🆕 Document Understanding (Document AI)
│ │ — Going beyond text extraction to understand document structure and semantics
│ │
│ ├── Layout-Aware Models
│ │ ├── LayoutLM (Microsoft) — BERT extended with 2D positional embeddings encoding
│ │ │ x,y coordinates of each word; pre-trained with Masked Visual Language Model
│ │ ├── LayoutLMv2 — Adds visual embeddings during pre-training (not just fine-tuning);
│ │ │ spatial-aware attention mechanism
│ │ ├── LayoutLMv3 — Unified text+image pre-training with word-patch alignment
│ │ ├── 🆕 UDOP — Unifying Document Processing; vision-text-layout unified model
│ │ └── 🆕 DocFormer — Multimodal Transformer fusing text, visual, and spatial features
│ │
│ ├── Document Tasks
│ │ ├── Form Understanding (FUNSD) — Extracting key-value pairs from forms
│ │ ├── Receipt/Invoice Extraction (SROIE) — Structured data from receipts
│ │ ├── Document Classification (RVL-CDIP) — Categorizing document types
│ │ ├── Table Detection — Finding table boundaries in document images
│ │ ├── Table Structure Recognition — Identifying rows, columns, and cells
│ │ ├── 🆕 Document VQA — Answering questions about document content
│ │ └── 🆕 Key Information Extraction (KIE) — Extracting structured fields
│ │
│ └── 🆕 Table Extraction Pipeline
│ ├── Table Detection (GTE Table) — Object detector finding table regions
│ ├── Cell Structure Recognition — Identifying cell boundaries and spanning cells
│ ├── Cell Content Extraction — OCR within each detected cell
│ └── Structure Reconstruction — Assembling extracted cells into structured data
│
└── 🆕 Evaluation Metrics
├── Word-level Accuracy / F1 — Correct words extracted vs. ground truth
├── CER (Character Error Rate) — Character-level accuracy for OCR
├── TEDS (Tree Edit Distance Score) — Measures table structure recognition quality
└── ANLS (Average Normalized Levenshtein Similarity) — Standard for Document VQA
video AI
Video AI
│
├── Video Understanding
│ │ — Analyzing and interpreting video content
│ │
│ ├── Video Classification / Action Recognition
│ │ ├── Two-Stream Networks — Separate spatial (RGB) and temporal (optical flow) streams
│ │ ├── C3D — 3D convolutions operating on video clips
│ │ ├── I3D — Inflated 2D convolutions (from ImageNet) to 3D for video
│ │ ├── SlowFast Networks — Dual pathway: slow (spatial) + fast (temporal motion)
│ │ ├── TimeSformer — Transformer with divided space-time attention for video
│ │ ├── ViViT — Video Vision Transformer processing tubelet tokens
│ │ └── 🆕 VideoMAE — Masked autoencoder pre-training adapted for video
│ │
│ ├── Temporal Action Detection
│ │ ├── Action Proposals — Detecting temporal segments containing actions
│ │ └── 🆕 ActionFormer — Transformer-based single-stage temporal action localization
│ │
│ ├── Video Object Tracking
│ │ ├── SORT / DeepSORT — Multi-object tracking using detection + Kalman filter + Re-ID
│ │ ├── 🆕 ByteTrack — Associates every detection box including low-confidence ones
│ │ └── SAM 2 Tracker — SAM extended to video with memory for consistent object tracking
│ │
│ └── Video-Language Models
│ ├── VideoBERT — Joint video-text pre-training with BERT-style objectives
│ ├── VideoLLaMA — LLM augmented with video understanding via temporal encoders
│ ├── 🆕 Video-ChatGPT — Video conversation model using spatiotemporal features
│ └── 🆕 Gemini Video — Native video understanding processing video as
│ continuous token stream (not choppy frame sampling)
│
├── Video Generation
│ │ — Creating video from text, images, or other video
│ │
│ ├── Core Techniques
│ │ ├── Video Diffusion Model (VDM) — Extends image diffusion to temporal dimension
│ │ ├── Spacetime Patches — Videos decomposed into spatial-temporal patch tokens
│ │ │ (analogous to text tokens in LLMs)
│ │ ├── Temporal Attention Layers — Added to image diffusion U-Net for frame coherence
│ │ ├── 🆕 Latent Video Compression — Encoder reduces video to compressed
│ │ │ latent representations before diffusion
│ │ └── 🆕 Motion Modeling — Predicting realistic physics-based object motion
│ │
│ └── Notable Video Generation Models
│ ├── Sora (OpenAI) — Diffusion Transformer on spacetime patches; up to 60s;
│ │ builds on DALL-E 3 recaptioning for prompt faithfulness
│ ├── Runway Gen-3 / Gen-4.5 — Up to 4K resolution; advanced camera controls
│ │ and scene coherence; 30+ creative tools
│ ├── Google Veo 2 / Veo 3 — Best overall quality (9.8/10); 4K output with
│ │ native audio sync
│ ├── Kling 3.0 — Up to 120 seconds; best value at scale
│ ├── Luma Dream Machine — Strong realistic motion and temporal consistency
│ ├── Pika — Beginner-friendly; 1080p; fast generation (45-90 seconds)
│ ├── 🆕 Wan (Alibaba) — Open-source video generation model
│ └── 🆕 LTX Video — Open-source 13B parameter video diffusion model
│
└── 🆕 Video Evaluation Metrics
├── FVD (Fréchet Video Distance) — Video-level FID measuring quality and diversity
├── Temporal Consistency Score — Measures frame-to-frame coherence
└── CLIP-T — Text-video alignment score using CLIP features