Nanochat — In-Depth Codebase Guide

What is Nanochat?
Key Achievement
Design Philosophy
Repository Structure
Model Architecture
Training Pipeline
Data Loading & Tokenization
Optimizer — MuonAdamW
Precision & Hardware
Evaluation System
Inference & Chat
Fine-Tuning (SFT & RL)
Configuration System — The Single Dial
Autoresearch — Autonomous Optimization
How to Run
Key File Reference
Glossary

What is Nanochat?

Nanochat is a minimal, full-stack LLM training harness that covers the entire pipeline from tokenization to a ChatGPT-like web UI — all in a single, hackable repository. It is designed for researchers and engineers who want to:

Train language models from scratch on a single GPU node (typically 8×H100)
Experiment with architecture and optimization ideas using a clean, readable codebase
Fine-tune and chat with the trained models immediately
Reproduce state-of-the-art efficiency results (train GPT-2-level models cheaply)

Think of it as “everything you need to go from raw text data → trained chat model → web UI” in ~15K lines of Python.

Key Achievement

Metric	Original GPT-2 (2019)	Nanochat (2026)
Training time	168 hours	1.65 hours (99 min)
Hardware	256 TPUv3	8× H100
Cost	~$43,000	~$48
Capability (CORE score)	0.2565	0.2626

Nanochat achieves GPT-2-level capability 100× faster and 900× cheaper than the original, through a combination of modern architecture, better optimizers, scaling laws, and hardware improvements.

Design Philosophy

1. Single Complexity Dial

The entire model is controlled by one parameter: --depth (number of transformer layers). Everything else — model width, number of attention heads, learning rate, batch size, training duration — is automatically computed from scaling laws.

--depth=12  →  small model for quick experiments (~5 min)
--depth=24  →  GPT-2 class model (~2 hours on 8×H100)
--depth=36  →  larger model for pushing capability

2. Compute-Optimal by Default

Nanochat uses Chinchilla-style scaling laws to automatically determine how many tokens to train on. The default ratio is 10.5 tokens per parameter, meaning a 124M parameter model trains on ~1.3B tokens.

3. Full Stack, Minimal Code

One repo covers: tokenizer training → data loading → pretraining → evaluation → fine-tuning → RL alignment → CLI chat → web UI. No external training frameworks needed.

4. Research-Friendly

Ideas tested at --depth=12 (5 minutes) transfer cleanly to --depth=24 (2 hours) because the scaling laws ensure all hyperparameters adjust proportionally.

Repository Structure

nanochat/
│
├── nanochat/                     # Core library (the "engine room")
│   ├── gpt.py                   # 🧠 GPT model architecture
│   ├── dataloader.py            # 📦 BOS-aligned BestFit data loader
│   ├── dataset.py               # 📥 Dataset download & management
│   ├── common.py                # 🔧 Utilities, dtype detection, DDP setup
│   ├── tokenizer.py             # ✂️ BPE tokenizer wrapper
│   ├── checkpoint_manager.py    # 💾 Save/load model checkpoints
│   ├── engine.py                # ⚡ Inference engine with KV cache
│   ├── optim.py                 # 🏋️ MuonAdamW optimizer
│   ├── core_eval.py             # 📊 DCLM CORE score evaluation
│   ├── loss_eval.py             # 📉 Bits-per-byte metric
│   ├── flash_attention.py       # ⚡ Flash Attention 3 / SDPA interface
│   ├── fp8.py                   # 🔢 FP8 training implementation
│   ├── execution.py             # 🐍 Python code execution (for agent tool use)
│   └── report.py                # 📋 Training report generation
│
├── scripts/                      # Entry points (what you actually run)
│   ├── base_train.py            # Pretrain a base language model
│   ├── base_eval.py             # Evaluate a pretrained model
│   ├── chat_sft.py              # Supervised fine-tuning
│   ├── chat_rl.py               # Reinforcement learning (DPO)
│   ├── chat_cli.py              # Interactive CLI chat
│   ├── chat_web.py              # Web UI chat (FastAPI)
│   ├── chat_eval.py             # Evaluate fine-tuned chat model
│   ├── tok_train.py             # Train a BPE tokenizer
│   └── tok_eval.py              # Evaluate tokenizer
│
├── tasks/                        # Evaluation task definitions
│   ├── common.py                # Base class + TaskMixture
│   ├── mmlu.py                  # Multiple-choice knowledge questions
│   ├── gsm8k.py                 # Grade-school math
│   ├── arc.py                   # Science reasoning
│   ├── humaneval.py             # Code generation
│   ├── smoltalk.py              # Conversational data
│   ├── spellingbee.py           # Spelling & counting tasks
│   └── customjson.py            # Custom JSON conversations
│
├── runs/                         # Ready-to-go training scripts
│   ├── speedrun.sh              # Full pipeline (train → finetune → eval)
│   ├── scaling_laws.sh          # Scaling law experiments
│   ├── miniseries.sh            # Train multiple model sizes
│   └── runcpu.sh                # CPU/MPS toy example
│
├── dev/                          # Development & research
│   ├── LOG.md                   # Detailed experiment log
│   ├── LEADERBOARD.md           # Time-to-GPT-2 leaderboard
│   └── *.ipynb                  # Analysis notebooks
│
└── tests/                        # Pytest tests

Data Flow Through the System

Raw Text (ClimbMix 400B on HuggingFace)
         │
         ▼
   ┌─────────────┐
   │  dataset.py  │  Downloads & caches tokenized data shards
   └──────┬──────┘
          │
          ▼
   ┌──────────────┐
   │ dataloader.py │  BOS-aligned bestfit packing into sequences
   └──────┬───────┘
          │
          ▼
   ┌────────────────┐
   │ base_train.py   │  Training loop (DDP, mixed precision, scaling laws)
   │   ├─ gpt.py     │  Forward pass through transformer
   │   ├─ optim.py   │  MuonAdamW backward pass & update
   │   ├─ core_eval  │  Periodic CORE score evaluation
   │   └─ checkpoint  │  Save model periodically
   └──────┬─────────┘
          │
          ▼
   ┌──────────────┐
   │  chat_sft.py  │  Fine-tune on task mixtures (MMLU, GSM8K, SmolTalk...)
   └──────┬───────┘
          │
          ▼
   ┌────────────────┐
   │  chat_rl.py     │  (Optional) DPO alignment
   └──────┬─────────┘
          │
          ▼
   ┌────────────────────────────┐
   │  chat_cli.py / chat_web.py │  Interactive chat (CLI or browser)
   │     └─ engine.py           │  KV-cached efficient generation
   └────────────────────────────┘

Model Architecture

File: nanochat/gpt.py

Nanochat implements a modern GPT-style transformer with several innovations stacked on top of the classic design. Here’s what makes it different from a vanilla transformer:

Core Transformer Structure

Input Token IDs
       │
       ▼
 ┌────────────────┐
 │ Token Embedding │  (vocab_size × model_dim)
 └───────┬────────┘
         │
         ▼  (repeated for each of `depth` layers)
 ┌───────────────────────────────────────────┐
 │              Transformer Block             │
 │                                           │
 │  ┌─ Smear Gate ──────────────────────┐    │
 │  │  Mix in previous token's embedding │    │
 │  └───────────────────────────────────┘    │
 │                                           │
 │  ┌─ Attention ───────────────────────┐    │
 │  │  QK-Norm → RoPE → GQA → FlashAttn │    │
 │  │  + Sliding Window (configurable)   │    │
 │  │  + Value Embeddings (alt layers)   │    │
 │  └───────────────────────────────────┘    │
 │                                           │
 │  ┌─ MLP ───────────────────���────────┐    │
 │  │  Linear → ReLU² → Linear          │    │
 │  └───────────────────────────────────┘    │
 │                                           │
 │  ┌─ Residual Connection ────────────┐    │
 │  │  resid_lambda × residual          │    │
 │  │  + x0_lambda × original_input     │    │
 │  │  - backout × mid_layer_residual   │    │
 │  └───────────────────────────────────┘    │
 └───────────────────────────────────────────┘
         │
         ▼
 ┌────────────────┐
 │  RMS LayerNorm  │
 └───────┬────────┘
         │
         ▼
 ┌────────────────────────────┐
 │ Output Projection (lm_head) │  → logits (softcapped to [-15, 15])
 └────────────────────────────┘

Key Architectural Innovations

1. Rotary Position Embeddings (RoPE)

Instead of learned position embeddings, RoPE encodes position information by rotating query and key vectors. This gives the model a natural sense of relative position and generalizes better to different sequence lengths.

# In CausalSelfAttention.forward():
q = apply_rotary_emb(q, cos, sin)  # Rotate queries by position
k = apply_rotary_emb(k, cos, sin)  # Rotate keys by position

2. QK Normalization

Queries and keys are normalized before the attention dot product. This stabilizes training and prevents attention logits from growing too large.

self.q_norm = nn.RMSNorm(head_dim)
self.k_norm = nn.RMSNorm(head_dim)

3. ReLU² Activation (Squared ReLU)

Instead of GELU, the MLP uses relu(x)². This creates sparser activations — most values are zero, and the few that are active are amplified. This improves both training efficiency and model quality.

def forward(self, x):
    x = self.c_fc(x)
    x = F.relu(x).square()  # ReLU² — sparse and sharp
    x = self.c_proj(x)
    return x

4. Per-Layer Learnable Scalars

Each layer has several learnable scalar parameters that control how information flows:

resid_lambda: Scales the residual connection (initialized 1.15 → 1.05 across layers). Controls how much of the layer’s output to keep.
x0_lambda: Blends the original input embedding back in at every layer. Prevents the model from “forgetting” the raw input.
smear_gate + smear_lambda: Mixes in the previous token’s embedding, giving the model bigram-like information at every layer.
backout_lambda: Subtracts a mid-layer residual stream to remove low-level features that are no longer useful in deeper layers.

5. Sliding Window Attention

Not every layer needs full-context attention. The window_pattern parameter (default "SSSL") controls which layers use short windows vs. full context:

S = short window (1/4 of context length)
L = full context length

Early layers handle local patterns (short window), later layers handle long-range dependencies (full context).

6. Value Embeddings (ResFormer)

Alternating layers add learnable “value embeddings” — position-dependent vectors added directly to the value tensor. This gives the model an additional channel to encode positional information.

7. Logit Softcap

Output logits are squeezed through 15 * tanh(logits / 15), capping them to [-15, 15]. This prevents extreme confidence in any single token and stabilizes training.

Automatic Configuration from `--depth`

When you set --depth=24, the model automatically computes:

model_dim = depth * aspect_ratio      # 24 * 64 = 1536
n_head = model_dim // head_dim        # 1536 // 128 = 12
n_kv_head = n_head                    # 12 (group-query attention)
mlp_dim = 4 * model_dim              # 6144
vocab_size = 32768                    # Fixed
sequence_len = 2048                   # Default context length

Training Pipeline

File: scripts/base_train.py

The Training Loop — Step by Step

1. Parse CLI arguments
2. Auto-compute hyperparameters from scaling laws
3. Initialize model on meta device (no memory until materialized)
4. Set up DDP (Distributed Data Parallel) across GPUs
5. Create MuonAdamW optimizer with per-group learning rates
6. Create data loader (BOS-aligned bestfit packing)
7. Training loop:
   a. Load batch of packed sequences
   b. Forward pass → compute cross-entropy loss
   c. Backward pass → compute gradients
   d. Optimizer step (Muon for matrices, AdamW for everything else)
   e. Periodically:
      - Evaluate validation loss (bits-per-byte)
      - Compute CORE score (22-task benchmark)
      - Generate text samples
      - Save checkpoint
8. Final evaluation & checkpoint

Scaling Laws — How Hyperparameters Are Auto-Computed

This is the most distinctive feature of nanochat. Given --depth, the system computes:

Number of tokens to train on:

num_tokens = target_param_data_ratio × num_scaling_params
           = 10.5 × ~124M
           ≈ 1.3B tokens

Batch size (in tokens):

total_batch_size = 2^19 × (target_depth / reference_depth)^0.383

This comes from the “Power Laws” paper — larger models benefit from larger batches.

Learning rate scaling (muP-style):

lr_scale = (768 / model_dim)^(-0.5)

All learning rates are multiplied by this factor, ensuring that wider models use appropriately adjusted learning rates.

Weight decay scaling:

Adjusted so that the "effective epoch length" (T_epoch) remains constant
when batch size changes.

Learning Rate Schedule

LR
 ▲
 │  ┌──────────────────┐
 │ /│                    \
 │/ │                     \
 │  │                      \
 │  │  warmup   constant    \  warmdown (65% of training)
 │  │  (40 steps)            \
 │  │                         \──── final LR = 5% of peak
 └──┴──────────────────────────────► Steps

The warmdown phase is unusually long (65% of training). This gradual cooldown is important for final model quality.

Gradient Accumulation

If your GPU doesn’t have enough memory for the full batch:

micro_batch_per_gpu = device_batch_size × sequence_length
full_batch = total_batch_size
accumulation_steps = full_batch / (micro_batch_per_gpu × num_gpus)

Gradients are accumulated over multiple forward passes before doing a single optimizer step.

Data Loading & Tokenization

Dataset: ClimbMix 400B

File: nanochat/dataset.py

Nanochat uses NVIDIA’s ClimbMix 400B dataset (karpathy/climbmix-400b-shuffle on HuggingFace), a curated mix of web text, code, and math. The data comes as pre-tokenized shards:

6543 shards available (each ~100M tokens)
Last shard reserved for validation
On-demand download with retry logic and caching

Tokenizer

File: nanochat/tokenizer.py

Type: BPE (Byte-Pair Encoding), GPT-4 style
Vocab size: 32,768 (2^15)
Special tokens: <|bos|>, <|user_start|>, <|user_end|>, <|assistant_start|>, <|assistant_end|>, <|python_start|>, <|python_end|>, <|output_start|>, <|output_end|>
Split pattern: Letters grouped, numbers max 2 digits per token, punctuation separate
Byte-level fallback for unknown characters

BOS-Aligned BestFit Packing

File: nanochat/dataloader.py

This is a clever data loading strategy that maximizes token utilization:

Traditional padding approach (wasteful):
[Doc1 tokens] [PAD PAD PAD PAD PAD PAD]  ← wasted space
[Doc2 tokens] [PAD PAD PAD]              ← wasted space
[Doc3 tokens] [PAD PAD PAD PAD]          ← wasted space

BOS-aligned BestFit packing:
[BOS Doc1 tokens Doc3-start ... crop]     ← 100% utilized
[BOS Doc2 tokens Doc5-fragment ... crop]  ← 100% utilized
[BOS Doc4 tokens ... crop]               ← 100% utilized

How it works:

Every sequence starts with a <|bos|> token
Documents are packed greedily — the largest document that fits is placed next
Remaining space is filled by cropping the next document
Result: 100% token utilization, ~35% of tokens are cropped (at sequence length 2048)

This means no compute is wasted on padding tokens.

Optimizer — MuonAdamW

File: nanochat/optim.py

Nanochat uses a dual optimizer strategy — different parameters get different optimizers:

The Split

Parameter Type	Optimizer	Why
Transformer weight matrices (Q, K, V, MLP)	Muon	Better for large matrices; uses orthogonalization
Embeddings	AdamW	Standard, works well for embeddings
Output projection (lm_head)	AdamW	Different learning rate needed
Per-layer scalars	AdamW	Tiny parameters, standard optimizer works fine

What is Muon?

Muon (Momentum + Orthogonalization) is an optimizer designed specifically for training neural network weight matrices. Instead of just following the gradient direction (like SGD) or maintaining running statistics (like Adam), Muon:

Maintains momentum (like SGD with momentum)
Orthogonalizes the update using the Polar Express method — this ensures the update has maximum “information content” per step

The orthogonalization step (5 iterations of a quintic formula) is the key innovation:

# Polar Express: find the nearest orthogonal matrix to the gradient
for _ in range(5):
    G = (1/6) * G * (15*I - Gt@G @ (10*I - 3*Gt@G))

Per-Group Learning Rates

Different parameter groups use very different learning rates:

Group	Learning Rate	Notes
Transformer matrices (Muon)	0.02	Main model parameters
Unembedding (lm_head)	0.008	Output projection
Token embeddings	0.3	Relatively high — embeddings need fast learning
Value embeddings	0.15	Moderate
Residual scalars	0.2	Per-layer control

All learning rates are scaled by (768/model_dim)^(-0.5) (muP scaling) to ensure consistent training dynamics across model sizes.

Momentum Schedule

Muon’s momentum isn’t constant — it follows a schedule:

Steps 0-400:     Ramp from 0.85 → 0.97 (warm up)
Steps 400-end:   0.97 during training, decay to 0.90 during warmdown

Precision & Hardware

Mixed Precision Training

File: nanochat/common.py

Nanochat auto-detects your GPU and chooses the best precision:

GPU	Precision	Notes
H100 (Hopper)	bfloat16 + optional FP8	Best performance
A100	bfloat16	Good performance
Older GPUs	float32	Fallback

How it works: Model weights are stored in full precision for the optimizer, but computations happen in reduced precision. A custom Linear layer handles the casting:

class Linear(nn.Linear):
    def forward(self, x):
        return F.linear(x, self.weight.to(x.dtype), self.bias)

FP8 Training

File: nanochat/fp8.py

For H100 GPUs, nanochat includes a minimal (~150 lines) FP8 implementation:

Tensorwise dynamic scaling: One scale factor per entire tensor (simpler than per-row/per-block)
Only applies to large matmuls: Dimensions must be ≥128 and divisible by 16
Uses torch._scaled_mm for cuBLAS FP8 kernels
~15% faster training with minimal quality loss

Flash Attention

File: nanochat/flash_attention.py

Provides a unified interface that tries, in order:

Flash Attention 3 (Hopper GPUs — fastest)
PyTorch SDPA (works everywhere)

Supports sliding window attention with configurable window sizes per layer.

Evaluation System

CORE Score

File: nanochat/core_eval.py

The primary benchmark is the DCLM CORE score — an ensemble of 22 evaluation tasks:

Multiple-choice tasks (accuracy):

MMLU (massive multitask), ARC-Easy, ARC-Challenge
Winogrande, PIQA, SociQA, HellaSwag
BoolQ, OpenBookQA, Copa

Generative tasks (F1 score):

SQuAD, TriviaQA, NaturalQuestions, DROP
AGIEval, Jeopardy, etc.

Format: Few-shot (typically 5 examples per task). The model sees examples and must answer in the same format.

Target: CORE score > 0.256525 = “GPT-2 level capability”

Bits Per Byte (BPB)

File: nanochat/loss_eval.py

A vocabulary-size-invariant loss metric:

BPB = total_nats / (ln(2) × total_bytes)

This accounts for the byte length of each token (a token representing “the” = 3 bytes), making it comparable across different tokenizers and vocab sizes.

Inference & Chat

Inference Engine

File: nanochat/engine.py

The engine uses a KV cache for efficient autoregressive generation:

Without KV cache (slow):
Step 1: Process [A]           → predict B
Step 2: Process [A, B]        → predict C
Step 3: Process [A, B, C]     → predict D   ← recomputes A, B each time!

With KV cache (fast):
Step 1: Process [A]           → predict B, cache K,V for A
Step 2: Process [B] + cache   → predict C, cache K,V for B
Step 3: Process [C] + cache   → predict D   ← only processes new token!

Features:

Pre-allocated KV cache tensors
Batched generation (multiple samples simultaneously)
Top-k sampling with temperature
Calculator tool (evaluates math expressions in model output)

Chat Interfaces

CLI (scripts/chat_cli.py):

python -m scripts.chat_cli -p "What is machine learning?"
python -m scripts.chat_cli  # Interactive mode

Web UI (scripts/chat_web.py):

python -m scripts.chat_web  # Opens browser at localhost:8000

Conversation format:

<|bos|><|user_start|>What is 2+2?<|user_end|><|assistant_start|>2+2 equals 4.<|assistant_end|>

Fine-Tuning (SFT & RL)

Supervised Fine-Tuning (SFT)

File: scripts/chat_sft.py

Takes a pretrained base model and fine-tunes it on task-specific data:

Base Model (good at predicting next token)
    │
    ▼  Fine-tune on conversations, Q&A, math problems...
    │
Chat Model (good at following instructions)

Task mixture example:

MMLU (knowledge) — 3 epochs
GSM8K (math reasoning) — 4 epochs
SmolTalk (conversation) — 1 epoch

Tasks are defined in the tasks/ directory. Each task provides:

Training examples as conversation turns
Evaluation metrics

Reinforcement Learning (DPO)

File: scripts/chat_rl.py

After SFT, optional Direct Preference Optimization (DPO) alignment:

Takes pairs of (preferred response, rejected response)
Trains the model to prefer the better response
Simpler and more stable than PPO/RLHF

Configuration System — The Single Dial

The Core Idea

Most LLM training frameworks have dozens of hyperparameters. Nanochat has one: --depth.

# This is all you need to specify:
torchrun --nproc_per_node=8 -m scripts.base_train --depth=24

Everything else is computed from scaling laws. But you can override anything:

Key CLI Arguments

Model size:

--depth          Number of transformer layers (THE dial)
--aspect-ratio   model_dim = depth × this (default: 64)
--head-dim       Attention head size (default: 128)
--max-seq-len    Context length (default: 2048)

Training horizon (pick one):

--num-iterations           Explicit step count
--target-flops             Fixed compute budget (for scaling law studies)
--target-param-data-ratio  Tokens-per-parameter ratio (default: 10.5)

Batch size & optimization:

--device-batch-size   Per-GPU batch size (reduce if OOM, default: 32)
--total-batch-size    Auto-computed if -1 (default)
--matrix-lr           Muon learning rate (default: 0.02)
--embedding-lr        Embedding learning rate (default: 0.3)

Precision:

--fp8                 Enable FP8 training (H100+ only)
NANOCHAT_DTYPE=...    Override auto-detection (bfloat16/float16/float32)

Evaluation:

--eval-every          Validation loss frequency (default: 250 steps)
--core-metric-every   CORE benchmark frequency (default: 2000 steps)
--sample-every        Text generation frequency (default: 2000 steps)

Autoresearch — Autonomous Optimization

One of the most fascinating aspects of this project: Claude (an AI) was given autonomous access to the codebase and ran two rounds of self-directed research to improve training efficiency.

Round 1 (March 9, 2026)

Claude ran autonomously for ~2 days
Explored architectural and optimization changes
Result: Reduced time-to-GPT-2 from 2.02 → 1.80 hours
All improvements found on --depth=12 (5 min experiments) generalized to --depth=24

Round 2 (March 14, 2026)

Introduced smear (bigram mixing) and backout (mid-layer residual subtraction)
Further hyperparameter tuning
Result: 1.80 → 1.65 hours (99 minutes)

Why This Works

The scaling law-based design means that improvements at small scale transfer to large scale. This creates a fast feedback loop:

Try idea on d12 (5 min) → Works? → Validate on d24 (2 hrs) → Ship it

How to Run

Prerequisites

Python 3.10+
PyTorch 2.0+ with CUDA
8× H100 GPUs (recommended) or any CUDA GPU for smaller experiments

Quick Start — Full Pipeline (~3 hours on 8×H100)

bash runs/speedrun.sh

This downloads data, trains a base model, fine-tunes it, and evaluates.

Step-by-Step

# 1. Train tokenizer (optional — pretrained one included)
python -m scripts.tok_train
 
# 2. Pretrain base model
torchrun --nproc_per_node=8 -m scripts.base_train --depth=24
 
# 3. Evaluate base model
torchrun --nproc_per_node=8 -m scripts.base_eval
 
# 4. Fine-tune for chat
torchrun --nproc_per_node=8 -m scripts.chat_sft
 
# 5. (Optional) RL alignment
torchrun --nproc_per_node=8 -m scripts.chat_rl
 
# 6. Chat with your model
python -m scripts.chat_cli -p "Hello!"
python -m scripts.chat_web  # Web UI

Small-Scale Experiments (any GPU)

# Quick experiment on a single GPU
python -m scripts.base_train --depth=6 --device-batch-size=8
 
# CPU/MPS toy example
bash runs/runcpu.sh

Scaling Law Studies

bash runs/scaling_laws.sh  # Trains models at various depths with fixed FLOP budgets

Key File Reference

File	Lines	Purpose
`nanochat/gpt.py`	~400	Model architecture — transformer blocks, attention, MLP, embeddings
`scripts/base_train.py`	~500	Main training loop — scaling laws, LR schedule, DDP, checkpointing
`nanochat/optim.py`	~350	MuonAdamW optimizer — dual strategy, Polar Express, momentum schedule
`nanochat/dataloader.py`	~300	BOS-aligned bestfit packing — 100% token utilization
`nanochat/engine.py`	~250	Inference with KV cache — efficient autoregressive generation
`nanochat/core_eval.py`	~400	CORE score — 22-task benchmark suite
`nanochat/common.py`	~200	Utilities — dtype detection, DDP setup, logging
`nanochat/fp8.py`	~150	FP8 training — minimal dynamic scaling implementation
`nanochat/dataset.py`	~150	Dataset management — download, cache, shard handling
`scripts/chat_sft.py`	~300	Supervised fine-tuning — task mixtures, multi-epoch training

Glossary

Term	Meaning
BOS	Beginning of Sequence — special token `<
BPB	Bits Per Byte — vocabulary-invariant loss metric
BPE	Byte-Pair Encoding — tokenization algorithm that merges frequent byte pairs
CORE score	DCLM evaluation metric — average across 22 NLP tasks
DDP	Distributed Data Parallel — PyTorch’s multi-GPU training
DPO	Direct Preference Optimization — alignment technique using preference pairs
FA3	Flash Attention 3 — Hopper-optimized attention kernel
FP8	8-bit floating point — reduces memory and increases speed on H100+
GQA	Group-Query Attention — shares KV heads across multiple query heads
KV Cache	Key-Value Cache — stores computed attention keys/values for fast generation
muP	Maximal Update Parameterization — scaling learning rates with model width
Muon	Momentum + Orthogonalization optimizer for weight matrices
ReLU²	Squared ReLU — `max(0, x)²`, creates sparse activations
RoPE	Rotary Position Embeddings — encodes position via rotation
SDPA	Scaled Dot-Product Attention — PyTorch’s built-in attention
SFT	Supervised Fine-Tuning — training on labeled conversation data
Softcap	Logit capping via `c * tanh(x/c)` — prevents extreme predictions
Chinchilla	Scaling law paper — optimal token-to-parameter ratio
ClimbMix	NVIDIA’s curated 400B token dataset (web + code + math)
Warmdown	Final phase of LR schedule where learning rate decays to near-zero

This guide was generated from the nanochat codebase at commit 5019acc.