Mechanistic interpretability

Mechanistic interpretability is a branch of AI research focused on understanding how neural networks actually work internally  not just what inputs and outputs they produce, but how their internal computations represent, transform, and combine information.

https://www.neuronpedia.org/

Anthropic’sInterpretability Research https://transformer-circuits.pub/

Goodfire is a research company using interpretability to understand, learn from, and design AI systems. Our mission is to build the next generation of safe and powerful AI https://www.goodfire.ai/

https://www.baseten.co/research/attention-based-attribution/#where-cosine-similarity-fails

Resources

• Reverse Engineering and Tracing internal thoughts of LLM
• Tracing the thoughts of a large language model
• https://www.lesswrong.com/posts/XGHf7EY3CK4KorBpw/understanding-llms-insights-from-mechanistic

https://github.com/chrishayuk/larql

The model IS the database. Query neural network weights like a graph database. No GPU required.

LARQL (Lazarus Query Language) is a revolutionary framework that treats neural network weights as a queryable graph database. The core premise: “The model IS the database.”

Instead of treating transformer models as black boxes, LARQL decompiles them into a queryable format called a vindex (vector index), then provides LQL (Lazarus Query Language) to browse, edit, inspect, and recompile neural network knowledge.

Model Weights (8 GB) → EXTRACT → Vindex (3-10 GB) → Query/Edit/Compile

---

Why You Need This

1. Interpretability without GPU: Query what the model knows without running inference
2. Knowledge Editing: Insert, delete, or update facts without retraining
3. No Fine-tuning: Lightweight JSON patch files (~10 MB per 1,000 facts) instead of 8 GB model retraining
4. Transparency: See exactly which neural features encode specific relations (capital, language, occupation, etc.)
5. Versioning: Stack patches, diff models, track knowledge changes

Example: Add “John Coyle lives in Colchester” to a Gemma model without retraining:

INSERT INTO EDGES (entity, relation, target)
    VALUES ("John Coyle", "lives-in", "Colchester");
-- Result: 1 edge inserted, model now knows this fact
INFER "Where does John Coyle live?" → "Colchester" (97.3%)

How It Works: End-to-End Flow

Phase 1: EXTRACT — Decompile Model → Vindex

Input: Model weights (Gemma 3 4B, 8 GB)
↓
EXTRACT MODEL "google/gemma-3-4b-it" INTO "gemma3-4b.vindex" WITH ALL;
↓
Output: Reorganized directory with queryable components

The Vindex Directory Structure:

gemma3-4b.vindex/
  ├─ gate_vectors.bin      # W_gate rows (KNN index, 3.3 GB)
  │                        # What features fire for a given token
  ├─ embeddings.bin        # W_embed matrix (token lookup, 2.5 GB)
  │                        # Token → dense vector
  ├─ down_meta.bin         # Feature output metadata (~2 MB)
  │                        # What does each feature output? (e.g., "Paris")
  ├─ attn_weights.bin      # Q, K, V, O matrices (3 GB, for INFER)
  ├─ up_weights.bin        # W_up matrices (3.3 GB, for COMPILE)
  ├─ lm_head.bin           # Output projection (1.3 GB, for COMPILE)
  ├─ index.json            # Metadata, layer bands, config
  ├─ tokenizer.json        # Model tokenizer
  ├─ feature_labels.json   # Probe-confirmed relation labels
  └─ relation_clusters.json # Discovered relation types

Extraction Levels:

Level	Files	Size (f16)	Capabilities
Browse	gate_vectors + embeddings + down_meta	~3 GB	DESCRIBE, WALK, SELECT
Inference	Above + attn_weights	~6 GB	+ INFER (full forward pass)
All	All files	~10 GB	+ COMPILE (back to model)

---

Phase 2: DESCRIBE — Query Model Knowledge

The Query:

USE "gemma3-4b.vindex";
DESCRIBE "France" VERBOSE;

Behind the Scenes:

1. Tokenize “France” → Token ID (e.g., 4235)
2. Look up embedding → embeddings.bin["France"] → 2560-dim vector
3. Gate KNN across layers (L14-27 = knowledge band):

   For each layer L:
     score = gate_vectors[L] @ embedding["France"]
     top_features = KNN(scores, top_k=10)
   

4. Annotate with down_meta:

   For each top feature:
     feature_meta = down_meta[layer][feature]
     output_token = feature_meta.top_token  # e.g., "Paris"
     confidence = feature_meta.c_score      # 0.0-1.0
   

5. Label with relations (from probe):

   If feature_meta.label == "capital" → [capital] → Paris
   If feature_meta.label == "language" → [language] → French
   

Output:

France
  Edges (L14-27):
    [capital]      → Paris                    9.2  L27      1x
    [language]     → français                14.7  L23      1x
    [continent]    → Europe                  14.4  L25      4x
    [—]            → Spain                   13.3  L18      1x

---

Phase 3: WALK — Feature Scan Without Attention

A fast, knowledge-only scan that shows which features activate for a token:

WALK "The capital of France is" TOP 10;

Flow:

1. Tokenize prompt → [The, capital, of, France, is]
2. Take last token embedding → embedding["is"]
3. KNN at each layer:

   For layer 14-27:
     scores = gate_vectors[layer] @ embedding["is"]
     top_features = top-10 features by score
     show relation labels + output tokens
   

Key difference from INFER: No attention, no context, just “what features fire for this embedding?”

Output (per layer):

L24:  F9515 (gate_score=105.7) → [capital] → Paris
L25:  F3842 (gate_score=94.4)  → [language] → French
L26:  F7261 (gate_score=83.1)  → [continent] → Europe

---

Phase 4: INFER — Full Forward Pass with Attention

The real inference engine with attention routing:

INFER "The capital of France is" TOP 5 COMPARE;

Complete Flow:

1. Tokenize & embed all tokens:

   Prompt: "The capital of France is"
   Tokens: [The, capital, of, France, is]
   Embeddings: [e_the, e_capital, ..., e_is]
   

2. Forward pass through 34 layers:

   For each layer L (0-33):
     
     A. ATTENTION HEAD:
        - Q = W_q @ residual
        - K = W_k @ history
        - V = W_v @ history
        - attention_output = softmax(Q·K^T / √d) @ V
        - residual += attention_output
     
     B. FFN (Using Walk FFN for efficiency):
        - gate_scores = W_gate @ residual  [0.98ms/layer via mmap]
        - top_features = top-K gate scores  [with INSERT edits]
        - ffn_out = Σ(top_features) * W_down_matrix
        - residual += ffn_out
   

3. Decode logits at final layer:

   logits = W_lm_head @ residual[last_token]
   probs = softmax(logits)
   top_5 = argsort(probs)[-5:]
   

4. Output:

   Predictions (walk FFN):
     1. Paris                  (97.91%)
     2. the                    (0.42%)
     3. a                      (0.31%)
   

Walk FFN Optimization (magic ingredient):

• Instead of dense FFN (W_up @ gate_out @ W_down), uses KNN gate lookup
• Reads W_down from mmap’d vindex (zero-copy BLAS)
• 517ms vs 535ms (actually faster than dense!)
• Uses only 3.5GB of weights, not 16.6GB

---

Phase 5: INSERT — Edit Knowledge

Add a fact without retraining:

INSERT INTO EDGES (entity, relation, target)
    VALUES ("John Coyle", "lives-in", "Colchester")
    AT LAYER 26
    CONFIDENCE 0.95
    ALPHA 0.30;

Behind the Scenes:

1. Auto-patch creation:

   - Base vindex remains readonly
   - Overlay patch created (never modifies base files)
   

2. Multi-layer constellation install (validated regime):

   The fact is not inserted at a single layer.
   Instead, a "constellation" of 8 synchronized layers is created:
   
   Layers installed: [22, 23, 24, 25, 26, 27, 28, 29]  (centered on L26)
   Alpha (strength): 0.30 per layer  (controls fact strength vs neighbor bleed)
   
   For each layer:
     - Create gate_vector: entity_embed * scale + relation_center * weight
     - Create down_vector: target_embed (initialized from embedding)
     - Update feature metadata: top_token = "Colchester", confidence = 0.95
   

3. Patch overlay:

   Edits.vlp (lightweight JSON):
   {
     "operations": [
       {
         "layer": 22,
         "feature": 9215,
         "type": "insert",
         "gate_vector": [...],
         "meta": {"top_token": "Colchester", "confidence": 0.95}
       },
       ... (7 more layers)
     ]
   }
   

4. Immediate effect:

   DESCRIBE "John Coyle"
   → "John Coyle" / [lives-in] → "Colchester"
   
   INFER "Where does John Coyle live?" → "Colchester" (98.2%)
   

---

Phase 6: COMPILE — Bake Patches Into New Vindex

Flatten all edits into a clean, standalone vindex:

COMPILE CURRENT INTO VINDEX "gemma3-4b-medical.vindex";

Flow:

1. Identify all inserted features (from patch overlay)
2. Rewrite down_weights.bin at inserted slots:

   - For each (layer, feature) in patch:
     - Write synthesized down vector to canonical down_weights.bin
     - Update down_meta.bin with new metadata
   

3. Hard-link unchanged files (on APFS, instant, zero-copy):

   - gate_vectors.bin → symlink from source
   - embeddings.bin → symlink from source
   - attn_weights.bin → symlink from source
   - (only down_weights.bin gets rewritten)
   

4. Result: New clean vindex with patches baked in, ~10 MB instead of 8 GB

Alternatively, compile back to standard model:

COMPILE CURRENT INTO MODEL "gemma3-4b-edited/" FORMAT safetensors;

→ Output is standard HuggingFace model, no special loader needed

---

Decoding/Feature Entry Flow: Detailed Example

Query: DESCRIBE "France"

Step-by-Step Execution:

1. TOKENIZATION
   Input: "France"
   tokenizer.tokenize("France") → [4235]  (single token)

2. EMBEDDING LOOKUP
   embeddings.bin[4235] → [0.23, -0.45, 0.89, ..., 0.12]  (2560 dims)
   
3. GATE KNN — Layer 14 (early knowledge)
   query = embedding_vec
   gate_matrix[L14] @ query → scores for all 10,240 features
   top_10 = [F2345, F3127, F4891, F1034, F2456, ...]
   
   KNN computation (0.008ms):
   - matmul: gate_vectors[L14] (10240×2560) @ query (2560)
   - sort by score
   - return top-10 indices + scores

4. FEATURE META LOOKUP (for each top-10 feature)
   down_meta[L14][F2345] → 
     {
       "top_token": "france",
       "top_token_id": 4250,
       "c_score": 0.88,
       "top_k": [
         {"token": "france", "logit": 5.1},
         {"token": "French", "logit": 3.2},
         {"token": "país", "logit": 1.8}
       ]
     }

5. LABEL LOOKUP (from probe)
   feature_labels.json["L14_F2345"] → "country"  (or null if not labeled)
   
6. ACROSS ALL LAYERS 14-27
   Repeat steps 3-5 for each layer
   Aggregate results by target token

7. MERGE & FORMAT
   Combine hits from all layers:
   "Paris" found at L27 (score 9.2) + L26 (score 7.1) → "Paris (9.2, L27)"
   
8. OUTPUT
   France
     Edges (L14-27):
       [capital]      → Paris              9.2  L27      1x
       [language]     → French            14.7  L23      1x
       [continent]    → Europe            14.4  L25      4x

---

Query Execution Architecture

┌─────────────────────────────────────────────┐
│           LQL Parser                        │
│  (DESCRIBE "France" VERBOSE)                │
└──────────────┬──────────────────────────────┘
               │
         ┌─────▼─────┐
         │   Parsed  │
         │    AST    │
         └─────┬─────┘
               │
    ┌──────────▼──────────┐
    │  Executor          │
    │ (route to backend)  │
    └─────────┬───────────┘
              │
    ┌─────────┴────────────┐
    │                      │
    ▼                      ▼
┌─────────────┐      ┌─────────────────┐
│ Vindex      │      │ Weight Backend  │
│ Backend     │      │ (live weights)  │
│ (fast KNN)  │      │                 │
│             │      │ (slower, reads  │
│ 0.98ms/L    │      │  from disk)     │
└─────────────┘      └─────────────────┘
    │
    └──────┬───────────────────┐
           │                   │
    ┌──────▼──────┐    ┌───────▼──────┐
    │ gate_knn    │    │ feature_meta │
    │ (per layer) │    │ (down_meta)  │
    └─────────────┘    └───────────────┘
           │                   │
           └─────────┬─────────┘
                     │
              ┌──────▼──────────┐
              │ Label Lookup    │
              │ (feature_labels)│
              └─────────────────┘
                     │
              ┌──────▼──────────┐
              │ Format Output   │
              │ (aggregation)   │
              └─────────────────┘

---

Architecture: 8 Crates

larql-models       ← Config, architecture traits, weight loading
    ↓
larql-vindex       ← Vindex lifecycle: extract, load, query, mutate, patch
    ↓
larql-core         ← Graph algorithms, merge, diff
larql-inference    ← Forward pass, BLAS attention, WalkFfn
    ↓
larql-lql          ← LQL parser, executor, REPL
    ↓
larql-server       ← HTTP/gRPC server (remote vindex access)
larql-cli          ← CLI commands

---

Performance: Why It’s Fast

Operation	Latency	Notes
Gate KNN (per layer)	0.008ms	BLAS matmul, mmap’d
Walk (34 layers)	0.3ms	Multi-layer scan
DESCRIBE (full)	33ms	Walk + label lookup
INFER (walk FFN)	517ms	With attention routing
Feature lookup	<1ns	Direct memory access

Walk FFN beats dense: 517ms vs 535ms

• Reads gate matrix from mmap (zero-copy BLAS)
• No GPU needed
• Only 3.5GB weights loaded

---

Three Extraction Levels

-- Level 1: Browse only (no GPU, fast queries)
EXTRACT MODEL "google/gemma-3-4b-it" 
    INTO "gemma3-4b.vindex"
    --level browse;  (default)
-- Size: ~3 GB
-- Enables: DESCRIBE, WALK, SELECT, EXPLAIN WALK
 
-- Level 2: Browse + Inference
EXTRACT MODEL "google/gemma-3-4b-it"
    INTO "gemma3-4b.vindex"
    WITH INFERENCE;
-- Size: ~6 GB
-- Enables: + INFER, EXPLAIN INFER, TRACE
 
-- Level 3: Full (all weights)
EXTRACT MODEL "google/gemma-3-4b-it"
    INTO "gemma3-4b.vindex"
    WITH ALL;
-- Size: ~10 GB
-- Enables: + COMPILE (recompile to model)

---

Summary Table

Component	Purpose	Speed	Storage
Vindex	Pre-extracted, queryable weights	0.3ms walk	3-10 GB
Gate KNN	”What features fire?“	0.008ms/layer	BLAS matmul
Down Meta	”What does each feature output?”	<1ns	~2 MB
Patch Overlay	Edit knowledge without retraining	—	~10 MB per 1K facts
Walk FFN	Efficient inference without GPU	517ms	3.5 GB
LQL	Query language	Interpreted	Parser in Rust

---

This is a complete reimagining of how we interact with neural networks—not as black boxes, but as queryable knowledge graphs that can be inspected, edited, and compiled without any training.