| Fine-Tuning (SFT) | Instruction Finetuning | Guides the model to follow user prompts accurately and helpfully across diverse task domains. |
| CoT Reasoning Finetuning | Teaches models to produce step-by-step reasoning traces rather than just final answers, improving accuracy on complex tasks. |
| Dialogue (Multi-turn) | Trains the model on transcripts of multiple turns to ensure it learns to maintain context across a conversation. |
| Knowledge Distillation | Uses a larger “teacher” model to generate labels or reasoning chains for a smaller “student” model to replicate. |
| Reinforcement Learning (RL) | PPO (Proximal Policy Optimization) | An online RL algorithm that uses a clipped objective to optimize policy while preventing catastrophic shifts in distribution. |
| DPO (Direct Preference Optimization) | Directly optimizes the policy from preference pairs (chosen vs. rejected) without requiring a separate reward model or critic. |
| GRPO (Group Relative Policy Optimization) | Eliminates the need for a separate critic model by estimating rewards from the average performance of a group of sampled outputs. |
| ORPO (Odds Ratio Preference Optimization) | Monolithic method that increases the likelihood of preferred responses relative to dispreferred ones using odds ratios. |
| OREO (Offline Reasoning Optimization) | An offline RL method that jointly trains a policy and value function to perform fine-grained credit assignment in multi-step tasks. |
| Test-Time Scaling (TTS) | Beam Search | Pruned breadth-first search that keeps the top N highest-probability partial sequences at each step. |
| Best-of-N Search | Generates N candidate outputs and selects the one with the highest score from a reward model or verifier. |
| Tree-of-Thoughts (ToT) | Structures reasoning as a search tree, allowing the model to explore branching paths and perform lookahead or backtracking. |
| Graph of Thoughts (GoT) | A graph-based structure that allows thoughts to be interconnected, merged (aggregated), and iteratively refined. |
| Monte Carlo Tree Search (MCTS) | Uses random simulations and rollouts to evaluate potential reasoning paths, balancing exploration and exploitation. |
| Self-Consistency Decoding | Samples multiple Chain-of-Thought (CoT) paths and selects the final answer that appears most frequently (majority vote). |
| Self-Refinement | An iterative process where the model generates an initial answer, critiques its own work, and then refines the response. |
| Efficiency & Adaptation | LoRA (Low-Rank Adaptation) | Injects trainable low-rank adapters into specific layers while leaving the rest of the base model weights frozen. |
| QLoRA | Combines 4-bit quantization with LoRA to enable high-quality fine-tuning on consumer-grade hardware. |
| GPTQ | A post-training quantization method that reduces model weights to 4 bits with minimal loss in quality. |
