Computer vision

The Core Pipeline

Every multimodal model follows roughly the same three-stage pipeline:

Encode – A vision encoder translates raw image pixels into embedding vectors
Align/Project – A connector bridges visual embeddings to the LLM’s token space
Fuse & Reason – The LLM reasons jointly over text and image tokens

Stage 1: Vision Encoder — Turning Pixels into Patches

The dominant approach is the Vision Transformer (ViT), which treats an image like a sentence:

The image is sliced into small fixed-size patches (e.g., 14×14 or 16×16 pixels)
Each patch is flattened into a vector just like a word token
Positional embeddings are added so the model knows spatial location of each patch
These patch embeddings flow through transformer layers where attention mechanisms allow each patch to learn relationships with other patches — e.g., a dog’s ear patch attends to the face patch to understand context
By the final layer, the model doesn’t see “brown pixels” but understands “golden retriever sitting on a beach”

Most models use CLIP (Contrastive Language-Image Pretraining) as the visual encoder, which was pretrained on billions of image-text pairs and already understands semantic alignment between vision and language.

Stage 2: The Connector / Alignment Layer

Raw visual features from ViT/CLIP can’t be directly fed into the LLM — they live in different vector spaces. Two dominant approaches bridge this gap:

Method	How it works	Used by
Linear Projection / MLP	A simple learnable layer maps image embeddings into LLM token space	LLaVA, early GPT-4V
Cross-Attention Adapter	Cross-attention layers inserted into LLM blocks; LLM attends to image tokens at each layer	LLaMA 3 Vision, Flamingo
Q-Former (Query Transformer)	A small transformer with learnable query tokens that extract compressed visual info	BLIP-2, InstructBLIP

The cross-attention approach preserves spatial structure better and scales well to high-resolution images and video. For example, LLaMA 3 Vision inserts cross-attention layers into every 4th transformer block of the LLM, allowing it to blend visual information deeply during text generation.

Stage 3: Multimodal Fusion & Reasoning

Once visual tokens are projected into the LLM space, they are concatenated with text tokens and fed through the LLM’s transformer layers together. Research shows a consistent stage-wise processing structure inside MLLMs:

Early layers → Visual grounding (where objects are, what they look like)
Middle layers → Cross-modal reasoning (connecting image content to the question)
Late layers → Answer decoding (generating the final text output)

How Gemini Does It Differently

Gemini takes a more native approach rather than stitching together separate models:

It was trained natively multimodal from scratch — text, images, audio, and video were all used during pre-training simultaneously, not added later as adapters
This creates richer conceptual connections: the model jointly learns what things look like, how they’re described, and how they behave in video
In a single inference pass, Gemini processes an image and a user’s text prompt together without needing a separate OCR or captioning step
This contrasts with models like early GPT-4V or LLaVA, which bolt a visual encoder onto a pre-trained text LLM

The 3 Major Architectural Approaches

Architecture	Connector	Example Models
Unified Token Sequence	Linear projection / MLP	LLaVA, Qwen-VL
Cross-Attention Fusion	Adapter cross-attn layers	LLaMA 3 Vision, Flamingo
Native Multimodal	No separate connector; trained end-to-end	Gemini, GPT-4o

What About OCR / Text in Images?

For reading text within images (screenshots, documents), models use their visual patch attention to recognize character shapes directly no separate OCR pipeline is required in modern models. Gemini can extract text from screenshots and reason about it natively in the same forward pass.

OpenSource models

https://moondream.ai/station (best and freee)

Book

Foundations of Computer Vision https://visionbook.mit.edu/

https://github.com/apple/ml-fastvlm