Reinforcement learning

1. Foundations

• What is Reinforcement Learning?
• RL vs Supervised Learning vs Unsupervised Learning
• Real-world examples of RL
• Agent–Environment interaction loop

2. Core Concepts & Terminology

• State
• Action
• Reward
• Environment
• Policy (deterministic vs stochastic)
• Episode vs continuous tasks
• Return (cumulative reward)
• Discount factor (γ)

3. Markov Decision Processes (MDP)

• Markov property
• MDP components: (S, A, R, P, γ)
• State transition probabilities
• Reward function
• Value of a state

4. Value Functions

• State-value function (V(s))
• Action-value function (Q(s, a))
• Bellman equations
• Optimal value functions

5. Dynamic Programming (Model-Based RL)

• Policy evaluation
• Policy improvement
• Policy iteration
• Value iteration
• Limitations of DP

6. Monte Carlo Methods

• Learning from complete episodes
• First-visit vs every-visit MC
• MC policy evaluation
• MC control
• Exploration strategies (ε-greedy)

7. Temporal Difference (TD) Learning

• TD vs Monte Carlo
• TD(0)
• SARSA (on-policy)
• Q-Learning (off-policy)
• Exploration vs exploitation trade-off

8. Function Approximation

• Why tabular methods don’t scale
• Linear function approximation
• Feature representation
• Generalization issues

9. Deep Reinforcement Learning

• Neural networks as function approximators
• Deep Q-Networks (DQN)
• Experience replay
• Target networks
• Common instability issues

10. Policy Gradient Methods

• Why value-based methods fail sometimes
• Policy gradients intuition
• REINFORCE algorithm
• Actor-Critic methods

11. Advanced Topics (Optional / Later)

• Proximal Policy Optimization (PPO)
• Trust Region Policy Optimization (TRPO)
• Model-based RL
• Multi-agent RL
• Exploration strategies
• Offline RL

12. Practical Skills

• OpenAI Gym / Gymnasium
• Implementing algorithms from scratch
• Debugging RL agents
• Evaluating performance
• Common pitfalls in RL

Agent–Environment Interaction Loop

This is the core loop of RL:

1. The agent observes the current state of the environment
2. The agent chooses an action
3. The environment changes state
4. The agent receives a reward
5. The cycle repeats

State → Action → Reward → Next State → Action → ...

The agent’s goal is to learn actions that lead to higher long-term rewards. At time step ( t ):

• State: $s_t\in \mathcal{S}$
• Action: $a_t\in \mathcal{A}$
• Reward: $r_{t+1}\in \mathbb{R}$
• Next state: $s_{t+1}$

The interaction is:

$$s_t\stackrel{a_t}{\to }(r_{t+1},s_{t+1})$$

2. Core Concepts & Terminology

State

A state represents the current situation of the environment.

Examples:

• Game board position in chess
• Robot’s position and speed
• Current screen in a video game

The state should contain enough information to make a good decision.

Action

An action is a choice the agent can make.

Examples:

• Move left / right / jump
• Buy / sell / hold a stock
• Accelerate / brake / turn

Actions affect how the environment changes.

Action space

• Discrete (where we have predefined steps or action)
• continuous (where we have infinte steps)

Reward

A reward is a numerical feedback signal from the environment.

• Positive reward → good action
• Negative reward (penalty) → bad action

Examples:

• +1 for winning a game
• −1 for crashing a robot
• Small reward for staying alive longer

Important: The agent’s goal is to maximize total reward, not just immediate reward.

Immediate Reward

Reward received right after an action: $r_{t+1}$

This tells you if the last action was good or bad.

Cumulative Reward (Return)

Total reward collected from now into the future. $G_t=r_{t+1}+r_{t+2}+r_{t+3}+\dots$

This tells you if a sequence of actions is good.

RL agents maximize cumulative reward, not immediate reward.

Discounted Cumulative Reward

Because:

• future is uncertain
• infinite time horizons exist
• immediate rewards are often more valuable

We discount future rewards.

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k+1}$$

Where:

• ( G_t ) = cumulative reward (return)
• ( \gamma \in [0,1] ) = discount factor
• ( r_{t+k+1} ) = reward at future time step

What the Discount Factor Really Does

Example Rewards $r=[1,;1,;1,;1]$

• If ( \gamma = 1 ) $G_0=1+1+1+1=4$
• If ( \gamma = 0.9 ) $G_0=1+0.9+0.9^2+0.9^3\approx 3.44$
• If ( \gamma = 0 ) $G_0=1$

-Larger ( \gamma ) → more future-oriented -Smaller ( \gamma ) → more short-term thinking we focus on getting immediate rewards

Return (Cumulative Reward)

The return is the total reward the agent collects over time.

Example:

Reward sequence: 1, 1, -1, 2
Return = 1 + 1 - 1 + 2 = 3

RL focuses on long-term success, not just the next reward.

Discount Factor (γ)

The discount factor (gamma) decides how much future rewards matter.

• γ close to 0 → focus on immediate rewards
• γ close to 1 → care about long-term rewards

Example:

• Short-term thinking: grab quick rewards
• Long-term thinking: sacrifice now for bigger future gains

Environment

The environment is everything outside the agent. It:

• Receives the agent’s action
• Updates the state
• Provides a reward

Examples:

• A game simulator
• The real world
• A physics engine

Policy (Deterministic vs Stochastic)

Accept state as input and return the action

A policy defines how the agent chooses actions.

• Deterministic policy: $a=\pi (s)$ Same state → same action

• Stochastic policy: $\pi (a\mid s)=P(A_t=a\mid S_t=s)$ Same state → action chosen with probabilities

Example:

• Deterministic: “Always turn left”
• Stochastic: “Turn left 70%, right 30%”

Policies control the agent’s behavior.

Episode vs Continuous Tasks

• Episode: A task with a clear start and end Example: A game that ends when you lose or win

• Continuous task: No natural ending Example: Robot balancing, stock trading

This affects how learning and rewards are handled.

Reinforcement Learning

Learn what to do by trying actions, observing consequences, and receiving rewards, with the goal of maximizing long-term reward.

Agent

The learner and decision-maker.

• Chooses actions
• Learns from experience
• Has a strategy (policy)

Environment

Everything outside the agent.

• Reacts to actions
• Changes state
• Gives rewards

Think: the world

State (S)

A snapshot of the environment at a moment.

Key idea:

A state contains all information needed to decide the next action

Example:

• Chess: board position
• Robot: joint angles + velocity
• Game: player position, enemies, score

If state is incomplete, learning becomes harder (POMDPs).

Action (A)

What the agent can do.

• Discrete: up/down/left/right
• Continuous: steering angle, torque

Actions change the state.

Reward (R)

A scalar feedback signal.

• Positive → good
• Negative → bad
• Zero → neutral

Critical insight:

The reward does NOT tell you how to act — only how good the outcome was

This is what makes RL hard.

Policy (π)

The agent’s behavior rule.

Formally:

π(a | s) = probability of taking action a in state s

Policy answers:

“If I see this state, what should I do?”

Policy can be:

• Deterministic
• Stochastic

“Just pick the action with the highest predicted reward.”

But that’s wrong.

Why?

• Some actions look bad now but lead to better futures
• Some actions look good but trap you

So the agent must balance:

• Exploitation → use what you know
• Exploration → try new things

This tradeoff is the heart of RL.

Markov decision process

The “Memoryless” Hack: A critical concept here is the Markov Property, which assumes the future depends only on the present state

If a system that is markov only if the future depends only on the present state not past etc.

• states (know where you are)
• actions (know what you can do )
• Reward (know if it good or bad)
• transitions (know what happens next) transition probablity

Discount Rate

Episodic and Continuous Tasks

Episodic Tasks are the tasks that have a final state. The task begins with a start state and ends with a final state, which is called an episode.

e.g. When you play a car racing game, the race starts with some initial state and when you complete the race, it ends at the final state. This complete race becomes an episode which is independent of another race/episode.

Continuous Tasks are the tasks that do not have a final state. In short, these tasks never end.

e.g. an automated stock trading agent will continuously interact with the stock market and make buy or sell decisions based on the current market conditions.

So we have agent which will start looking for gain the reward but for a task could potentially go on forever, an agent might achieve an “infinite score,“which is impossible to optimize. without knowing our target it hard to solve.

To solve this we introduce Discount Rate (γ)

This is a number between 0 and 1. A reward received now is worth more than a reward received later

Policy

A policy is a commitment to how the agent will behave in every possible situation.

Formally:

$$\pi (a\mid s)=P(\text{take action }a\text{ in state }s)$$

Two types:

Deterministic

$$\pi (s)=a$$

Always the same move.

Stochastic

\pi(a \mid s) \in $$0,1$$, \quad \sum_a \pi(a \mid s) = 1

Useful when:

• Exploration matters
• The environment is adversarial
• Randomization is optimal

Key insight:

The policy is not what evaluates actions. It is what chooses actions.

Evaluation comes next.

Why values must exist at all

Suppose you only had a policy.

Ask:

“Why is this action good?”

The policy can’t answer.

So you invent a function that predicts long-term return.

That function is the value function.

State-value V(s): “How good is it to be here?”

Definition:

$$V^{\pi }(s)={\mathbb{E}}_{\pi }\left[G_t\mid S_t=s\right]$$

Meaning:

If I start in state (s) and follow policy π forever, how much reward do I expect?

This tells you:

• Whether a state is safe or dangerous
• Whether being here is promising or hopeless

But it does not tell you:

• Which action to take now

That’s a fatal limitation.

Action-value Q(s, a): “What if I do THIS?”

So you go one level deeper.

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[G_t\mid S_t=s,A_t=a\right]$$

Interpretation:

If I’m in state (s), take action (a) now, and then follow π forever, how good is that?

This is a counterfactual evaluation:

• “What happens if I choose this move?”

This is why Q-values feel powerful.

The key relationship (this is the breakthrough)

The relationship between V and Q is:

$$V^{\pi }(s)=\sum _a\pi (a\mid s)Q^{\pi }(s,a)$$

Meaning:

A state is valuable if the actions you tend to take from it are valuable.

And the optimal versions satisfy:

$$V^{\ast }(s)=\underset{a}{\max }{Q}^{\ast }(s,a)$$

This equation is the heart of control in RL.

Bellman Equation

The Bellman Equation is the “master formula” that connects the present to the future. It states that the value of your current position is the immediate reward plus the discounted value of where you land next

action space

• continius space (infinite no of step self drive car)
• diseceste space

Task

• epdisodic task
• continous task

ploicy

• deterministic
• stochastic policy
• Exploration and exploitation

State value function action value function

Apporach to reinforncement

• Model based
• Model free based
• policy based
  • reinforce
  • ppo
  • tRPRP
• value based
  • Q learning
  • deep q learning
  • double deep q learning

policy gradients actor critic proximal policy optimization

monte carlo temporal difference

RL 6 questions

Every RL algorithm must answer six design questions.

Think of them like six engineering design choices.

Question	What it controls
1	What the agent sees and can do
2	How the agent explores
3	Whether it learns a world model
4	Whether it evaluates states
5	How it estimates future reward
6	How it keeps learning stable

1. Perception & Agency: What do I see and do?

Before learning can happen, we must define the boundaries. This is the Interface.

• The State (s): This is the information the agent uses to make a decision. Crucially, from first principles, the agent doesn’t “know” what the numbers mean. To a robot, a camera feed is just a matrix of numbers.
• The Action (a): This is the lever the agent can pull.
  • Discrete: Turning left or right (Switch).
  • Continuous: Pushing an accelerator 12.5% (Slider).

Mental Model: Think of the agent as being in a dark room with a small window (Observation) and a joystick (Action). It has no manual; it only knows what happens next by moving the stick.

2. Motivation: Why do I move?

If an agent just sits there, it learns nothing. We need Heat.

• Extrinsic Reward: “Eat the cookie, get +1 point.”
• Intrinsic (Curiosity): “I’ve never been in this corner of the room before; that’s interesting, get +0.1 point.”
• Exploration (Entropy): To prevent the agent from just doing the same “safe” thing forever (Exploitation), we force it to be “messy” or “random” early on.

3. The “Dream” vs. Reality: Where do I practice?

This is the divide between Model-Free and Model-Based.

• Model-Free: You learn by doing. You touch a stove, it burns, you don’t do it again. You don’t need to understand thermodynamics to learn not to touch the stove.

• Model-Based: You try to learn the “Laws of Physics” of the environment (P(s′∣s,a)). Once you have a “World Model,” you can simulate thousands of scenarios in your “head” (the GPU) without ever touching the real stove.

4. Evaluation: How do I measure “Good”?

This is the “Value” vs. “Policy” debate.

• Value-Based (The Critic): “How much is this specific spot on the board worth?” If I know the value of every spot, I just move to the highest-value spot.

• Policy-Based (The Actor): “I don’t care how much this spot is worth; I just know that in this situation, I should jump.” It maps State → Action directly.

• Actor-Critic: The Actor tries to move, and the Critic stands behind it saying, “That move was better/worse than I expected.”

5. Temporal Credit Assignment: How do I plan for the future?

If you win a game of Chess on move 50, was move 5 the “winning” move?

• Monte Carlo: Wait until the very end. Look back and say, “We won, so every move we made must have been okay.” (High variance, but honest).

• Temporal Difference (TD): Update your guess based on another guess. “I thought I’d have a 50% chance of winning, but after move 5, I now think I have a 60% chance. Move 5 was good!” (Low variance, but can be biased).

6. Stability: How do I stay grounded?

RL is notoriously unstable because the “data” changes as the agent gets better.

• Experience Replay: Saving old memories so the agent doesn’t forget how to walk while it’s learning how to run.

• Trust Regions (PPO/GRPO): Telling the agent, “You can change your mind, but don’t change it by more than 10% at a time.” This prevents “catastrophic forgetting.”

The ultimate goal of the agent is to find a Policy (a strategy mapping states to actions) that maximizes the Return (the cumulative future rewards, which are often reduced over time by a “discount factor” to prioritize immediate success)

Resources

• https://www.reinforcenow.ai/ [best]
• https://www.youtube.com/watch?v=SupFHGbytvA&list=PL_iWQOsE6TfVYGEGiAOMaOzzv41Jfm_Ps
• https://www.youtube.com/watch?v=EvHRQhMX7_w
• https://github.com/anakin87/llm-rl-environments-lil-course
• Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci
• https://www.youtube.com/watch?v=o6l6tJQgUg4 [best]
• https://rlhfbook.com

https://nrehiew.github.io/blog/sft_rl_opd/

Tools

• MinT: RL Infrastructure for Experiential Intelligence
• Tinker is a training API for builders
  • https://tinker-docs.thinkingmachines.ai/