Assume the model is picking the next word and these are the probabilities after softmax:
| Token | Probability |
|---|---|
| A | 0.40 |
| B | 0.30 |
| C | 0.15 |
| D | 0.10 |
| E | 0.05 |
Total = 1.00 (100%)
Temperature — “How sharp or flat is the distribution?”
Temperature rescales the logits before softmax, but mentally you can think of it like this:
Temperature = 1.0 (normal)
Nothing changes.
A = 0.40 B = 0.30 C = 0.15 D = 0.10 E = 0.05
This is the baseline.
Temperature < 1 (example: 0.5 → more confident)
The highest probability becomes even more dominant.
A ≈ 0.60 B ≈ 0.25 C ≈ 0.09 D ≈ 0.04 E ≈ 0.02
Now token A is very likely → more predictable, less creative.
If temperature → 0 → it becomes almost 100% A → like greedy.
Temperature > 1 (example: 1.5 → more random)
The distribution flattens:
A ≈ 0.28 B ≈ 0.25 C ≈ 0.20 D ≈ 0.15 E ≈ 0.12
Now even unlikely words have a real chance → more creative, more risky.
Temperature just controls:
How much the model “trusts” its best guess
- Low T → “Play it safe”
- High T → “Take risks”
Top-K — “Only look at the best K options”
Top-K means:
Ignore all tokens except the top K most probable
Let’s say K = 3
We keep only:
| Token | Probability |
|---|---|
| A | 0.40 |
| B | 0.30 |
| C | 0.15 |
Drop:
- D (0.10)
- E (0.05)
Now renormalize so total = 1 again:
Total = 0.40 + 0.30 + 0.15 = 0.85
New probabilities:
A = 0.40 / 0.85 = 0.47 B = 0.30 / 0.85 = 0.35 C = 0.15 / 0.85 = 0.18
And we sample only from A, B, C
What Top-K really does:
- Puts a hard cut-off
- No matter what, small tokens like E can NEVER be picked
- Fixed count of tokens
- If K=1 → same as greedy
Top-P (Nucleus) — “Keep tokens until total probability reaches P”
Instead of a fixed number of tokens, Top-P uses a probability threshold.
Sort tokens by probability (already sorted):
| Token | Prob | Cumulative |
|---|---|---|
| A | 0.40 | 0.40 |
| B | 0.30 | 0.70 |
| C | 0.15 | 0.85 |
| D | 0.10 | 0.95 |
| E | 0.05 | 1.00 |
Let’s say P = 0.85
We include tokens until cumulative ≥ 0.85
So we keep:
A (0.40) B (0.30) C (0.15) → total = 0.85
We drop D and E.
This looks similar to Top-K = 3 here, BUT Important difference
If the distribution changes to:
A = 0.70 B = 0.15 C = 0.05 D = 0.05 E = 0.05
Top-P = 0.85 would select:
A (0.70) + B (0.15) = 2 tokens only
Whereas Top-K = 3 would always pick 3 tokens even if they are bad.
When model picks next token:
- Get logits
- Apply temperature
- Apply softmax
- Apply top-K filter
- Apply top-P filter
- Renormalize
- Randomly sample one token
That token is added to the sentence. Then the whole process repeats for the next word.