Reinforcement learning

1. Foundations

• What is Reinforcement Learning?
• RL vs Supervised Learning vs Unsupervised Learning
• Real-world examples of RL
• Agent–Environment interaction loop

2. Core Concepts & Terminology

• State
• Action
• Reward
• Environment
• Policy (deterministic vs stochastic)
• Episode vs continuous tasks
• Return (cumulative reward)
• Discount factor (γ)

3. Markov Decision Processes (MDP)

• Markov property
• MDP components: (S, A, R, P, γ)
• State transition probabilities
• Reward function
• Value of a state

4. Value Functions

• State-value function (V(s))
• Action-value function (Q(s, a))
• Bellman equations
• Optimal value functions

5. Dynamic Programming (Model-Based RL)

• Policy evaluation
• Policy improvement
• Policy iteration
• Value iteration
• Limitations of DP

6. Monte Carlo Methods

• Learning from complete episodes
• First-visit vs every-visit MC
• MC policy evaluation
• MC control
• Exploration strategies (ε-greedy)

7. Temporal Difference (TD) Learning

• TD vs Monte Carlo
• TD(0)
• SARSA (on-policy)
• Q-Learning (off-policy)
• Exploration vs exploitation trade-off

8. Function Approximation

• Why tabular methods don’t scale
• Linear function approximation
• Feature representation
• Generalization issues

9. Deep Reinforcement Learning

• Neural networks as function approximators
• Deep Q-Networks (DQN)
• Experience replay
• Target networks
• Common instability issues

10. Policy Gradient Methods

• Why value-based methods fail sometimes
• Policy gradients intuition
• REINFORCE algorithm
• Actor-Critic methods

11. Advanced Topics (Optional / Later)

• Proximal Policy Optimization (PPO)
• Trust Region Policy Optimization (TRPO)
• Model-based RL
• Multi-agent RL
• Exploration strategies
• Offline RL

12. Practical Skills

• OpenAI Gym / Gymnasium
• Implementing algorithms from scratch
• Debugging RL agents
• Evaluating performance
• Common pitfalls in RL

Agent–Environment Interaction Loop

This is the core loop of RL:

1. The agent observes the current state of the environment
2. The agent chooses an action
3. The environment changes state
4. The agent receives a reward
5. The cycle repeats

State → Action → Reward → Next State → Action → ...

The agent’s goal is to learn actions that lead to higher long-term rewards. At time step ( t ):

• State: $s_t\in \mathcal{S}$
• Action: $a_t\in \mathcal{A}$
• Reward: $r_{t+1}\in \mathbb{R}$
• Next state: $s_{t+1}$

The interaction is:

$$s_t\stackrel{a_t}{\to }(r_{t+1},s_{t+1})$$

2. Core Concepts & Terminology

State

A state represents the current situation of the environment.

Examples:

• Game board position in chess
• Robot’s position and speed
• Current screen in a video game

The state should contain enough information to make a good decision.

Action

An action is a choice the agent can make.

Examples:

• Move left / right / jump
• Buy / sell / hold a stock
• Accelerate / brake / turn

Actions affect how the environment changes.

Action space

• Discrete (where we have predefined steps or action)
• continuous (where we have infinte steps)

Reward

A reward is a numerical feedback signal from the environment.

• Positive reward → good action
• Negative reward (penalty) → bad action

Examples:

• +1 for winning a game
• −1 for crashing a robot
• Small reward for staying alive longer

Important: The agent’s goal is to maximize total reward, not just immediate reward.

Immediate Reward

Reward received right after an action: $r_{t+1}$

This tells you if the last action was good or bad.

Cumulative Reward (Return)

Total reward collected from now into the future. $G_t=r_{t+1}+r_{t+2}+r_{t+3}+\dots$

This tells you if a sequence of actions is good.

RL agents maximize cumulative reward, not immediate reward.

Discounted Cumulative Reward

Because:

• future is uncertain
• infinite time horizons exist
• immediate rewards are often more valuable

We discount future rewards.

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

Where:

• ( G_t ) = cumulative reward (return)
• ( \gamma \in [0,1] ) = discount factor
• ( r_{t+k+1} ) = reward at future time step

What the Discount Factor Really Does

Example Rewards $r=[1,;1,;1,;1]$

• If ( \gamma = 1 ) $G_0=1+1+1+1=4$
• If ( \gamma = 0.9 ) $G_0=1+0.9+0.9^2+0.9^3\approx 3.44$
• If ( \gamma = 0 ) $G_0=1$

-Larger ( \gamma ) → more future-oriented -Smaller ( \gamma ) → more short-term thinking we focus on getting immediate rewards

Return (Cumulative Reward)

The return is the total reward the agent collects over time.

Example:

Reward sequence: 1, 1, -1, 2
Return = 1 + 1 - 1 + 2 = 3

RL focuses on long-term success, not just the next reward.

Discount Factor (γ)

The discount factor (gamma) decides how much future rewards matter.

• γ close to 0 → focus on immediate rewards
• γ close to 1 → care about long-term rewards

Example:

• Short-term thinking: grab quick rewards
• Long-term thinking: sacrifice now for bigger future gains

Environment

The environment is everything outside the agent. It:

• Receives the agent’s action
• Updates the state
• Provides a reward

Examples:

• A game simulator
• The real world
• A physics engine

Policy (Deterministic vs Stochastic)

Accept state as input and return the action

A policy defines how the agent chooses actions.

• Deterministic policy: $a=\pi (s)$ Same state → same action

• Stochastic policy: $\pi (a\mid s)=P(A_t=a\mid S_t=s)$ Same state → action chosen with probabilities

Example:

• Deterministic: “Always turn left”
• Stochastic: “Turn left 70%, right 30%”

Policies control the agent’s behavior.

Episode vs Continuous Tasks

• Episode: A task with a clear start and end Example: A game that ends when you lose or win

• Continuous task: No natural ending Example: Robot balancing, stock trading

This affects how learning and rewards are handled.

RL 6 questions

Every RL algorithm must answer six design questions.

Think of them like six engineering design choices.

Question	What it controls
1	What the agent sees and can do
2	How the agent explores
3	Whether it learns a world model
4	Whether it evaluates states
5	How it estimates future reward
6	How it keeps learning stable

1. Perception & Agency: What do I see and do?

Before learning can happen, we must define the boundaries. This is the Interface.

• The State (s): This is the information the agent uses to make a decision. Crucially, from first principles, the agent doesn’t “know” what the numbers mean. To a robot, a camera feed is just a matrix of numbers.
• The Action (a): This is the lever the agent can pull.
  • Discrete: Turning left or right (Switch).
  • Continuous: Pushing an accelerator 12.5% (Slider).

Mental Model: Think of the agent as being in a dark room with a small window (Observation) and a joystick (Action). It has no manual; it only knows what happens next by moving the stick.

2. Motivation: Why do I move?

If an agent just sits there, it learns nothing. We need Heat.

• Extrinsic Reward: “Eat the cookie, get +1 point.”
• Intrinsic (Curiosity): “I’ve never been in this corner of the room before; that’s interesting, get +0.1 point.”
• Exploration (Entropy): To prevent the agent from just doing the same “safe” thing forever (Exploitation), we force it to be “messy” or “random” early on.

3. The “Dream” vs. Reality: Where do I practice?

This is the divide between Model-Free and Model-Based.

• Model-Free: You learn by doing. You touch a stove, it burns, you don’t do it again. You don’t need to understand thermodynamics to learn not to touch the stove.

• Model-Based: You try to learn the “Laws of Physics” of the environment (P(s′∣s,a)). Once you have a “World Model,” you can simulate thousands of scenarios in your “head” (the GPU) without ever touching the real stove.

4. Evaluation: How do I measure “Good”?

This is the “Value” vs. “Policy” debate.

• Value-Based (The Critic): “How much is this specific spot on the board worth?” If I know the value of every spot, I just move to the highest-value spot.

• Policy-Based (The Actor): “I don’t care how much this spot is worth; I just know that in this situation, I should jump.” It maps State → Action directly.

• Actor-Critic: The Actor tries to move, and the Critic stands behind it saying, “That move was better/worse than I expected.”

5. Temporal Credit Assignment: How do I plan for the future?

If you win a game of Chess on move 50, was move 5 the “winning” move?

• Monte Carlo: Wait until the very end. Look back and say, “We won, so every move we made must have been okay.” (High variance, but honest).

• Temporal Difference (TD): Update your guess based on another guess. “I thought I’d have a 50% chance of winning, but after move 5, I now think I have a 60% chance. Move 5 was good!” (Low variance, but can be biased).

6. Stability: How do I stay grounded?

RL is notoriously unstable because the “data” changes as the agent gets better.

• Experience Replay: Saving old memories so the agent doesn’t forget how to walk while it’s learning how to run.

• Trust Regions (PPO/GRPO): Telling the agent, “You can change your mind, but don’t change it by more than 10% at a time.” This prevents “catastrophic forgetting.”

The ultimate goal of the agent is to find a Policy (a strategy mapping states to actions) that maximizes the Return (the cumulative future rewards, which are often reduced over time by a “discount factor” to prioritize immediate success)

Resources

• https://www.reinforcenow.ai/ [best]
• https://www.youtube.com/watch?v=SupFHGbytvA&list=PL_iWQOsE6TfVYGEGiAOMaOzzv41Jfm_Ps
• https://www.youtube.com/watch?v=EvHRQhMX7_w
• https://github.com/anakin87/llm-rl-environments-lil-course
• Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci
• https://www.youtube.com/watch?v=o6l6tJQgUg4 [best]
• https://rlhfbook.com

https://nrehiew.github.io/blog/sft_rl_opd/

Tools

• MinT: RL Infrastructure for Experiential Intelligence
• Tinker is a training API for builders
  • https://tinker-docs.thinkingmachines.ai/