Gemma 4: End-to-End Architecture Walkthrough
A note on sourcing: Gemma 4 was released on April 2, 2026 — after my reliable training cutoff (end of Jan 2026). Everything below is reconstructed from Google’s model cards, the official developer guide, and a widely-cited independent visual breakdown, verified via live web search rather than recalled from memory. Where the underlying research paper isn’t public, I say so explicitly rather than guessing.
0. The Model Family (context you need before the pipeline makes sense)
Gemma 4 isn’t one model — it’s five checkpoints sharing a common design philosophy:
| Model | Type | Total / Active Params | Modalities In | Context |
|---|---|---|---|---|
| E2B | Dense (+ Per-Layer Embeddings) | ~2B effective | Text, Image, Audio | 128K |
| E4B | Dense (+ Per-Layer Embeddings) | ~4B effective | Text, Image, Audio | 128K |
| 12B “Unified” | Dense, encoder-free | 12B | Text, Image, Audio | 128K |
| 26B A4B | Mixture-of-Experts | 26B total / 4B active | Text, Image | 256K |
| 31B | Dense | 31B | Text, Image | 256K |
Two genuinely different architectural patterns exist inside this family:
- The “classic” encoder-based pipeline (E2B, E4B, 26B A4B, 31B) — dedicated vision/audio encoders feed a decoder-only backbone.
- The “unified” encoder-free pipeline (12B) — raw pixels and waveforms are projected directly into the backbone with no dedicated encoder at all.
I’ll walk through the classic pipeline first (it’s the conceptual foundation), then show how the 12B variant collapses it.
Phase 1 — Raw Input → Modality-Specific Tokens
This is where text, images, and audio stop being “media” and become sequences of vectors.
1a. Text → Tokens
Standard subword tokenization against a large vocabulary (262,144 tokens in the small models). Each token ID is looked up in an embedding table, producing a dense vector at the model’s hidden dimension (e.g., 1536 for E2B).
1b. Image → Patches (Vision Transformer front-end)
- Adaptive resize: unlike a classic ViT that forces a square crop, Gemma 4 resizes the image to preserve its original aspect ratio, padding only where a clean 16×16-pixel patch grid doesn’t fit exactly.
- Patchify: the image is chopped into 16×16 pixel patches — the same “an image is a sequence of words” trick from the original ViT paper.
- Variable resolution via token budget: the developer picks a soft token budget (70 / 140 / 280 / 560 / 1120). This budget determines how aggressively the image is downscaled before patchifying — a 280-token budget allows up to 2,520 raw patches (9× the budget), because…
- 3×3 spatial pooling: neighboring patches are merged in 3×3 blocks (via averaging) until the sequence is pooled down to roughly the requested token budget. This is a deliberate compute/quality dial — higher budgets keep more spatial detail (good for OCR, charts, object detection), lower budgets are cheaper and sufficient for coarse scene understanding or high-frame-rate video.
- 2D RoPE for position: because the patch grid’s shape changes per image, a single 1D position index (“patch #4”) is meaningless — patch 4 is the center of a 3×3 grid but not of a 2×4 grid. Gemma 4 solves this by splitting each patch embedding in half and applying rotary position encoding twice — once using the patch’s width coordinate, once using its height coordinate — so positional meaning survives arbitrary aspect ratios.
1c. Audio → Frames (Conformer front-end, E2B/E4B/12B only)
- Mel-spectrogram extraction: raw 16kHz waveform is converted into a 2D time-vs-frequency representation.
- Chunking: spectrogram slices are grouped into overlapping chunks.
- Convolutional downsampling: two 2D convolutional layers compress the chunks into a shorter sequence of continuous “soft tokens” (there’s no discrete audio vocabulary — these are dense vectors, not IDs).
Phase 2 — Modality Encoding & Projection into a Shared Embedding Space
This is the heart of “how does fusion actually work” — and the honest answer is: there is no cross-attention between modalities. Gemma 4 uses early fusion, not cross-attention fusion. Here’s the mechanism:
- Vision encoder: the pooled, position-encoded patch sequence passes through a dedicated Vision Transformer (150M params on E2B/E4B, 550M params on the larger models). Output: one embedding per visual token, but in the vision encoder’s native dimensional space — not yet compatible with the language model.
- Audio encoder: the downsampled audio chunks pass through a Conformer (a Transformer encoder augmented with a convolutional module, well-suited to local acoustic patterns) — again producing embeddings in its own native space.
- Linear projection + RMSNorm: both the vision and audio encoder outputs are passed through a small trained linear projection layer, followed by RMSNorm, to reshape them into exactly the hidden dimension and value distribution the language backbone expects. This projection is trained jointly with the rest of the model, so the projected visual/audio tokens land in a region of embedding space the LLM can actually interpret alongside text tokens.
- Concatenation, not cross-attention: the projected image tokens, audio tokens, and text tokens are simply concatenated into one single sequence in the order they appear in the prompt (e.g.,
<text><image tokens><text><audio tokens><text>). From this point on, the model doesn’t know or care which tokens came from which modality — they’re just vectors in a shared sequence.
Why this design, and why it matters: because fusion happens before the transformer rather than through a separate cross-attention module, every attention layer in the backbone is automatically a multimodal attention layer — text tokens attend to image tokens and vice versa using the exact same QKV mechanism used for text-to-text attention. There’s no separate “vision-language cross-attention block” bolted on. This is architecturally simpler than dual-tower designs with explicit cross-attention adapters, at the cost of requiring the projection layers to be trained very carefully so the modalities are actually aligned in space.
Phase 3 — The Core Transformer Backbone
Once everything is a token in one sequence, it flows through a stack of decoder blocks. This is where Gemma 4 diverges most from Gemma 3.
3a. Interleaved Local/Global Attention
Every few layers use sliding-window (local) attention — a token only attends to a fixed recent window — interleaved with occasional global attention layers where a token attends to the entire context.
- Small models (E2B/E4B): 4 local layers : 1 global layer, local window = 512 tokens.
- Larger models (26B A4B / 31B): 5 local layers : 1 global layer, local window = 1024 tokens.
- Change from Gemma 3: the final layer is now always global attention (Gemma 3’s ratio sometimes left the last layer as local, which hurt long-range coherence right before output).
Why bother with local attention at all? Global (full) attention scales quadratically with sequence length; sliding-window attention is roughly linear, and stacking several local layers still lets information propagate across the full sequence indirectly (like a relay), while periodic global layers “refresh” true long-range awareness.
3b. Making the Expensive Global Layers Cheaper
Global attention layers are the bottleneck because they must cache Keys/Values for the entire context. Gemma 4 applies three tricks specifically to global layers:
- Grouped-Query Attention (GQA) at 8:1 — 8 query heads share a single KV head in global layers (vs. 2:1 in local layers), sharply cutting KV-cache size.
- Doubled Key dimensionality — to compensate for the quality loss from more aggressive query grouping.
- K = V — in global layers only, the Key and Value projections are set equal, effectively halving the cache to just a “K-cache.”
3c. p-RoPE (Partial Rotary Position Embeddings)
Standard RoPE rotates every pair of dimensions in the Query/Key vectors by a frequency that decreases across the vector — high-frequency pairs encode precise position, low-frequency pairs stay closer to the original (positionless) semantic content. The problem: over very long contexts, low-frequency rotations still accumulate and can misalign tokens that are far apart, and they add positional noise to what’s meant to be a channel for semantic information. Gemma 4’s fix (used only in global layers, where context length is largest) is to zero out rotation entirely on the low-frequency dimensions — with p=0.25, only the first 25% of dimension-pairs get any positional signal at all, leaving the rest purely semantic.
3d. Normalization and Feed-Forward
Pre- and post-RMSNorm bracket both the attention and feed-forward sub-blocks (consistent with Gemma 3). The feed-forward path is where the two big architectural forks in the family diverge:
Dense variants (E2B, E4B, 12B, 31B): every layer runs one standard feed-forward network (FFN).
MoE variant (26B A4B): every layer instead runs a Mixture-of-Experts block:
- 128 total experts, of which a router activates 8 per token based on learned routing probabilities.
- A shared expert (3× the size of a normal expert) is always active for every token, intended to hold general-purpose knowledge, while the 8 routed experts specialize on token-specific content.
- The routed experts’ outputs are weighted by their router probability and summed with the shared expert’s output.
- Net effect: 26B parameters live in memory (sparse/total parameters), but only ~4B parameters’ worth of compute actually runs per token (active parameters) — hence “A4B.” This is why a 26B-class model can run at speeds close to a 4B dense model.
3e. Per-Layer Embeddings (PLE) — small models only
E2B/E4B add a second, much smaller embedding lookup (256-dim, vs. 1536/2560-dim main embeddings) per layer. At the start of inference, each input token’s per-layer embeddings are fetched once (not recomputed per step), gated, projected up to the main hidden size, normalized, and added back into that layer’s output — effectively “reminding” each layer what the original token was, without the cost of extra full-size parameters. Crucially, this large per-layer lookup table lives in flash storage rather than VRAM, since it’s a cheap one-time lookup rather than something that needs repeated matrix compute — this is the actual reason E2B/E4B can run inside tight mobile RAM budgets despite nominally having more total parameters than “2B” or “4B” implies (hence “E” for “effective,” not total, parameters).
Phase 4 — From Final Hidden State to Output Tokens
- LM Head: the final layer’s hidden state is projected back to vocabulary-size logits (a matmul against essentially the same matrix used for input embedding lookup).
- Sampling: standard decoding controls (temperature, top-p/top-k) select the next token from that distribution — Gemma 4 doesn’t publicly document a novel sampler here; this stage is conventional.
- Speculative decoding via Multi-Token Prediction (MTP) drafters: alongside every main (“target”) checkpoint, Google released a matching, much smaller “drafter” model (e.g., ~76M parameters for the E2B drafter). Instead of the large model generating strictly one token per forward pass, the flow is:
- The drafter autoregressively proposes several candidate next tokens very cheaply.
- The target model verifies all candidates in a single parallel forward pass.
- Every candidate the target model agrees with is accepted outright; at the first disagreement, the target model substitutes its own token and the rest of the draft is discarded.
- The drafter is boosted by three tricks: it consumes the target model’s final-layer activations (concatenated with its own token embeddings) instead of starting cold each round; it cross-attends to the target model’s already-computed KV-cache rather than building its own; and (E2B/E4B only) its LM head first predicts a cluster of likely tokens before narrowing to individual tokens, avoiding a full 262K-way softmax on a tiny model.
- Output is text-only across every Gemma 4 variant — even though input can be text+image+audio, generation never emits image or audio tokens.
The Alternate Path: Gemma 4 12B “Unified” (Encoder-Free)
The 12B model deletes Phase 2 entirely. Instead of routing images/audio through dedicated ViT/Conformer encoders:
- Vision: 48×48 pixel patches are projected straight into the LLM’s embedding space with a single matrix multiplication (a 35M-parameter “vision embedder” replacing what was a 27-layer, 550M-parameter transformer). A factorized X/Y coordinate lookup attaches 2D spatial position directly at this projection step.
- Audio: raw 16kHz waveform is sliced into 40ms frames (640 raw float samples each) and linearly projected straight into the same embedding space — no Conformer at all.
- Everything — text, vision, audio — is then processed by literally the same decoder-only transformer stack (same design as the 31B dense model’s backbone).
The tradeoff being made explicit here: dedicated encoders (ViT, Conformer) are pretrained, arguably richer feature extractors, but they add latency, separate parameter budgets, and — because they’re normally frozen — block full end-to-end fine-tuning. The encoder-free design costs some raw per-modality feature quality in exchange for lower latency and letting a LoRA or full fine-tune pass touch the entire multimodal pipeline in one go, since there’s no frozen sub-network left out of the loop.
Full Pipeline at a Glance (classic encoder-based variants)
TEXT ──► tokenizer ──► embedding lookup ─────────────────┐
│
IMAGE ──► adaptive resize ──► 16x16 patchify ──► 3x3 pool │
──► 2D RoPE ──► ViT (150M/550M) ──► linear proj │──► concatenated
──► RMSNorm │ token sequence
│ │
AUDIO ──► mel-spectrogram ──► chunk ──► conv downsample │ ▼
──► Conformer encoder ──► linear proj ──► RMSNorm┘ Decoder-only
Transformer
backbone:
• interleaved local/
global attention
• GQA (2:1 local,
8:1 global)
• K=V (global only)
• p-RoPE (global only)
• Dense FFN or
MoE (128 experts,
8 routed + 1 shared)
• (+ PLE on E2B/E4B)
│
▼
LM Head → logits
│
MTP drafter proposes,
target verifies (spec.
decoding) → sampling
│
▼
Text output tokens
Comparative Analysis
A few caveats up front, in the interest of giving you an accurate picture rather than a confident-sounding one: Llama 4’s and GPT-4o’s full architectural specifics are partially public (Meta has published more detail than OpenAI), and Anthropic has not published Claude’s internal architecture at all — so any comparison involving Claude is necessarily incomplete on that side, and I’m not going to fabricate details Anthropic hasn’t disclosed. What follows is what’s actually verifiable.
Where Gemma 4’s choices are distinctive
- Open-weights, Apache 2.0, self-hostable — this is the single biggest practical differentiator from GPT-4o and Claude, which are API-only with no downloadable weights. Gemma 4 also moved off Google’s older custom Gemma license onto plain Apache 2.0, removing usage-cap and acceptable-use-policy friction that had made some enterprises hesitant about earlier Gemma generations. Llama 4 is Gemma 4’s closest peer on this axis (also open-weight), though under Meta’s own license rather than Apache 2.0.
- Dense and MoE variants of the same generation, at multiple sizes — Llama 4 also ships both dense and MoE variants, so this isn’t unique to Gemma, but Gemma 4’s spread (2B-effective up through 31B dense, with a 26B/4B-active MoE in between) is unusually broad, deliberately covering phone → laptop → single-GPU → server in one family.
- The encoder-free 12B “Unified” variant is the more genuinely novel piece. Mainstream multimodal LLMs (including, as far as is publicly known, GPT-4o and Gemini) use dedicated vision/audio encoders feeding a language backbone. Collapsing that into direct linear projection of raw patches/waveform is an unusual bet, and its stated payoff is lower multimodal latency and the ability to fully fine-tune the whole stack (including “vision” and “audio” processing) in one pass, rather than needing to separately handle frozen encoder weights.
- Aggressive KV-cache engineering (K=V, 8:1 GQA, p-RoPE, and local/global interleaving) is squarely aimed at making long-context, on-device inference viable on constrained hardware — a priority that shows up much more if your target includes phones and 16GB-VRAM laptops than if your target is exclusively data-center GPUs.
- Native Multi-Token-Prediction drafters shipped as first-class artifacts, not just an internal serving trick — this is somewhat unusual as a public, downloadable speculative-decoding companion model, though Meta has also published MTP-related work and various providers use speculative decoding internally without publishing the drafter.
- Variable visual token budget (70–1120) exposes resolution-vs-cost as an explicit dial to the developer, which is less commonly surfaced this directly in competing APIs (GPT-4o/Claude typically abstract image tokenization away entirely, at the cost of the caller having less control).
Where it’s reportedly behind
Independent write-ups covering the release note Gemma 4 trails some competing open models — specifically Qwen 3.5 27B — on agentic coding benchmarks like SWE-bench Verified, and that every Gemma 4 variant is text-output-only, with no native speech or image generation, unlike GPT-4o’s native audio-out capability.
Bottom line
The clearest, most defensible claim is about deployment model, not raw intelligence: Gemma 4 trades some of the “everything handled server-side, zero-ops” convenience of GPT-4o/Claude/Gemini for full weight access, on-device viability, and fine-tunability — and its architecture (encoder-free variant, aggressive cache compression, per-layer embeddings) is engineered specifically in service of that tradeoff. Raw capability comparisons against closed frontier models are much harder to make responsibly, since two of the three comparison points (GPT-4o, Claude) don’t have public architecture specs to verify claims against.