K.Boopathi
BlogNotesSecond BrainProjectsHire MeYour reading file0 pts

Whisper

Jan 12, 20264 min read

Paper: https://cdn.openai.com/papers/whisper.pdf

Whisper must:

  1. Convert raw audio into text
  2. Work on long audio (minutes to hours)
  3. Handle unknown languages
  4. Avoid cutting words or sentences at fixed boundaries

The design choices you see (30‑second windows, timestamps, seek logic, prompts) are all answers to these constraints.

Keep this question in mind:

How do we transcribe unbounded audio using a fixed-size neural network without losing continuity?

1. High-Level Pipeline

Audio → Log‑Mel Spectrogram → 30s Window
      → Encoder (once per window)
      → Decoder (token-by-token loop)
      → Tokens + Timestamps
      → Smart Seek → Next Window

Everything else is detail.

2. Entry Point: transcribe()

The transcribe() function orchestrates the entire process.

2.1 Audio Normalization

  • Audio is converted into a log‑mel spectrogram
  • If audio < 30s → pad with silence to exactly 30s

Why?

  • The model is trained on fixed-size inputs
  • Variable-length inputs complicate batching and attention masks

3. Language Detection (Before Transcription)

Whisper must decide which tokenizer and decoding mode to use.

3.1 Key Insight

Whisper does not use a separate language classifier.

Instead:

It forces the main decoder to reveal the language using a single-token trick.

3.2 The One‑Token Trick

  1. Take the first 30s spectrogram
  2. Feed the decoder only:
<|startoftranscript|>
  1. Run one forward pass
  2. Mask out all tokens except language tokens:
<|en|>, <|fr|>, <|de|>, ...
  1. Pick the highest‑probability survivor
mask[list(tokenizer.all_language_tokens)] = False
logits[:, mask] = -np.inf
language = logits.argmax(dim=-1)

Why this works:

  • Whisper is trained to always emit a language token immediately after <|sot|>

4. Windowing Strategy (Why 30 Seconds?)

Whisper processes audio in overlapping 30‑second windows.

Why 30s?

  • Fits GPU memory
  • Long enough for semantic context
  • Short enough for fast iteration

But fixed windows introduce a dangerous problem:

What if a word or sentence crosses the 30s boundary?

This is solved later using timestamps + smart seek.

5. The Encoder — “The Ears”

The Encoder converts sound → meaning.

5.1 Input

  • Source: audio.py → log_mel_spectrogram()
  • Shape:
(batch, 80, 3000)
  • 80 = Mel frequency bins
  • 3000 = 30s ÷ 10ms frames

This is an image of sound.

5.2 Compression (The Zipper)

Inside AudioEncoder.forward():

  1. Conv1d #1
    • Expands channels → n_state (e.g. 512)
  2. GELU non-linearity
  3. Conv1d #2 (stride = 2)
    • Halves time resolution

Result:

(batch, 512, 1500)

Interpretation:

  • Each encoder token ≈ 20ms of audio
  • Audio becomes 1500 audio tokens

5.3 Positional Embedding

  • Added to tell the model when sounds happened
  • Without it, attention would be time‑agnostic

5.4 Self‑Attention Blocks

  • Multiple ResidualAttentionBlocks
  • Every audio moment can attend to every other moment

Example:

  • Noise at 5s can influence word interpretation at 6s

5.5 Encoder Output

Final result:

AudioFeatures
Shape: (batch, 1500, n_state)

This is meaning without words.

6. The Decoder “The Writer”

The Decoder converts meaning → text.

It runs autoregressively (one token at a time).

6.1 Decoder Inputs

  1. AudioFeatures (from Encoder)
  2. Tokens generated so far

Initial tokens look like:

[<|sot|>, <|en|>, <|transcribe|>]

6.2 Decoder Anatomy

Each decoding step contains:

  1. Self‑Attention (text memory)
  2. Cross‑Attention (audio lookup)
  3. MLP + projection → logits

6.3 Cross‑Attention (The Critical Bridge)

x = x + self.cross_attn(
    self.cross_attn_ln(x),  # text query
    xa,                     # audio features
)[0]

Mental model:

  • Text asks: “Given what I’ve written so far, which audio moment matters next?”
  • Audio replies with the relevant slice

This is how words align to sound.

7. Token‑by‑Token Decoding Example

Phrase: “Welcome to New York”

Step 1 — “Welcome”

  • Attention focuses on audio tokens ~0–50
  • Output: Welcome

Step 2 “ to”

  • Attention shifts slightly forward
  • Output: to

Step 3 “ New”

  • Attention moves deeper into the window
  • Output: New

Step 4 “ York”

  • Continues until phrase ends

Note:

  • Whisper tokens include spaces

8. Vocabulary & Tokenization

  • Vocabulary stored in binary file:
whisper/assets/gpt2.tiktoken
  • Loaded in tokenizer.py
  • Maps base64‑encoded strings → token IDs

Includes:

  • Words
  • Subwords
  • Spaces
  • Timestamps
  • Control tokens

9. The Hard Problem: 30‑Second Boundary Cuts

Naively splitting audio would cut words:

Ama|zing

Whisper avoids this using timestamps + smart seek.

10. Timestamps The Safety Mechanism

Whisper emits timestamps like:

<|28.00|> Hello world <|29.20|>

Important constraint:

The model only emits timestamps for completed phrases

If a word starts but doesn’t finish before 30s:

  • No closing timestamp is emitted

11. Smart Seek “Predict, Then Rewind”

After decoding a window:

  1. Find the last valid timestamp
  2. Move the cursor (seek) back to that point
last_timestamp = tokens[pos] - tokenizer.timestamp_begin
seek += last_timestamp * input_stride

Result:

  • Window 1: 0–30s → text ends at 28s
  • Window 2: starts at 28s → captures full word

No word is lost.

12. Context Memory Across Windows

Even though audio overlaps, text must remain coherent.

Whisper solves this with prompt carryover:

  • Previous text is injected as a hidden prefix

  • New window decoding is conditioned on past text

decode_options["prompt"] = all_tokens[prompt_reset_since:]

This preserves:

  • Grammar
  • Capitalization
  • Sentence flow

13. Final Mental Model

Whisper behaves like this:

“Listen to 30 seconds → understand → write until confident → rewind slightly → continue with memory.”

Graph View