Transformer-based LLMs (like GPT-3) achieved strong NLP results but initially failed at logical and mathematical reasoning tasks. The challenge of solving problems like GSM8K sparked research into methods to improve multi-step reasoning.
Chain-of-Thought (CoT): The Breakthrough
Adding simple prompts like “Let’s think step by step” led to massive accuracy gains (e.g., GPT-3 175B: 15.6% → 46.9% on GSM8K). CoT showed that guiding models to output intermediate steps enables emergent reasoning abilities.
Three-Stage Framework
LLM reasoning can be formalized as:
- Generate – Prompting the model to create the steps needed to solve the problem
- Evaluate – Prompting the model to assess the quality or correctness of the generated steps
- Control – Managing the sequential decision process, including how steps are searched or executed
Step Generation Methods
- Hand-written Prompts: Manual CoT or Zero-shot CoT (“Let’s think step by step”).
- External Knowledge: Self-ask uses external tools/search.
- Model-Generated Prompts: Auto-CoT and Complexity-based prompting let models create optimal reasoning prompts.
→ Trend: From manual → automated prompting for scalability.
Step Evaluation Methods
When LLMs are prompted to perform reasoning steps, they perform a sequence of steps and predict multiple tokens. Performing a sequence of steps makes them sensitive to mistakes and vulnerable to error accumulation (logical, factual, ethical, or otherwise) . Several methods have been developed to prevent error accumulation
-
Self-Assessment: Self-consistency (ensemble of reasoning paths) and self-verification improve reliability.
- Self-consistency asks the LLM to simply perform the same query multiple times, and takes the majority-vote of the answers, or decoding paths. Self-consistency works since complex reasoning problems typically allow different reasoning paths that lead to the correct answer
-
Tool-Based Validation: Program-of-Thought and PAL translate reasoning into executable code checked by interpreters.
-
External Model Validation: Critic/refiner models or physics-based grounding (e.g., Say-can) enhance correctness.
→ Key idea: Use code and external tools for objective step checking.
Step Control Methods
- Greedy: Sequential CoT or Least-to-Most prompting.
- Ensemble: Multiple reasoning paths merged.
- RL & Search: Tree-of-Thoughts (ToT), ReAct, Reflexion integrate LLMs with external algorithms for exploration and feedback.
→ Control mechanisms link symbolic search/RL with neural LLM reasoning.
Resources Multi-Step Reasoning with Large Language Models, a Survey
Resources
- Understanding Reasoning LLMs
- The Rise Of Reasoner Models: Scaling Test-Time Compute
- Test-Time Compute in Generative AI: An AI Atlas Report
Note:
What an LLM actually is doing
- At base, an LLM predicts the next token given the entire prompt + tokens generated so far.
- The model’s parameters and attention layers compute probabilities over the vocabulary; sampling/decoding picks the next token from that distribution.
- Nothing in the basic mechanism is a symbolic planner everything is statistical pattern completion over token sequences learned from data.
Why “Chain-of-Thought” (CoT) works
-
CoT changes the distribution of plausible next tokens.
If the prompt asks for intermediate steps, the model is conditioned to continue with tokens that read like reasoning steps. These tokens often include explicit calculations, sub-problems, and checks that make the final answer more likely to be correct. -
It exposes latent structure. Large models implicitly store patterns for multi-step solutions in their weights; CoT just encourages the model to surface that latent computation as text.
-
It reduces the need for a single giant jump. Instead of attempting to jump from problem → final answer in one short token(s), the model performs a sequence of smaller, easier prediction steps (token-by-token for each intermediate step). Each small step is an easier prediction, so the overall chain is more robust.