Reinforcement Learning

 

Chapter 15 — Reinforcement Learning

---

1. Definition

Reinforcement Learning (RL) is a type of Machine Learning in which an Agent learns by interacting with an Environment through trial and error.

Instead of learning from labeled data (Supervised Learning) or finding hidden patterns (Unsupervised Learning), the agent learns by receiving Rewards and Penalties for its actions.

The goal of the agent is to maximize the total reward over time by learning the best sequence of actions.

In simple words:

Reinforcement Learning teaches a computer to make better decisions by rewarding good actions and penalizing bad actions.

---

Example

Imagine teaching a dog to sit.

• If the dog sits correctly → Give it a treat (Reward).

• If the dog does not sit → No treat or a correction (Penalty).

Over time, the dog learns that sitting leads to rewards, so it repeats that behavior.

Reinforcement Learning works in the same way.

---

Another Example

Suppose a robot is trying to reach a charging station.

Robot → Move Left → Wrong Direction → Penalty

Robot → Move Right → Closer → Reward

Robot → Move Right → Charging Station → Big Reward

After many attempts, the robot learns the best path to the charging station.

---

2. Purpose

The main purpose of Reinforcement Learning is to teach an agent to make the best decisions by maximizing rewards.

Unlike other Machine Learning methods, Reinforcement Learning focuses on decision-making rather than prediction or clustering.

It is commonly used when a machine must learn through experience.

Businesses and researchers use Reinforcement Learning for:

• Robot Navigation

• Self-Driving Cars

• Game Playing

• Stock Trading

• Traffic Signal Optimization

• Industrial Automation

• Resource Allocation

• Personalized Recommendations

---

3. When to Use

Use Reinforcement Learning when:

• The problem involves making a sequence of decisions.

• The system can learn through trial and error.

• Rewards or penalties can be defined.

• The environment changes based on the agent's actions.

• There is no fixed dataset with correct answers.

Real-Life Applications

• Self-Driving Cars

• Chess and Board Games

• Video Game AI

• Robotics

• Warehouse Automation

• Drone Navigation

• Smart Traffic Lights

• Investment Strategies

• Dynamic Pricing

• Recommendation Systems

---

4. Major Reinforcement Learning Algorithms

The two most popular beginner-friendly Reinforcement Learning algorithms are:

Algorithm	Purpose
Q-Learning	Learns the best action by maximizing future rewards. It is an off-policy algorithm.
SARSA	Learns from the action actually taken by the agent. It is an on-policy algorithm.

These two algorithms are considered the foundation of Reinforcement Learning and are the best starting point before learning advanced methods like Deep Q-Networks (DQN) or Proximal Policy Optimization (PPO).

---

Summary

Section	Description
Definition	Reinforcement Learning is a Machine Learning approach where an agent learns through trial and error using rewards and penalties.
Purpose	To learn the best sequence of actions that maximizes long-term rewards.
When to Use	Use when a system must make decisions over time and can learn by interacting with an environment.
Major Algorithms	Q-Learning and SARSA.

---

Chapter 16 — Q-Learning

---

1. Definition

Q-Learning is a Reinforcement Learning algorithm that teaches an Agent how to choose the best action in each situation (called a State) by learning through trial and error.

It learns by receiving Rewards and Penalties from the environment.

The letter Q stands for Quality.

The Q-value represents how good a particular action is in a particular state.

In simple words:

Q-Learning helps a computer learn the best action to take in every situation by remembering which actions give the highest rewards.

---

Example

Suppose a robot wants to reach a treasure.

Start

↓

Move Right

↓

Reward = +10

↓

Move Right

↓

Reward = +10

↓

Treasure

↓

Reward = +100

If the robot moves into a wall,

Move Left

↓

Wall

↓

Reward = -20

After many attempts, the robot learns:

• Which moves give rewards.

• Which moves give penalties.

• Which path reaches the treasure fastest.

---

2. Purpose

The main purpose of Q-Learning is to find the best action for every state so that the total future reward is as large as possible.

Unlike Supervised Learning, Q-Learning does not require labeled data.

Instead, it learns by interacting with the environment repeatedly.

It is commonly used for:

• Robot Navigation

• Game Playing

• Path Finding

• Warehouse Robots

• Self-Driving Cars

• Resource Allocation

• Traffic Signal Control

---

3. When to Use

Use Q-Learning when:

• The problem involves decision-making.

• An agent can interact with an environment.

• Rewards and penalties can be defined.

• The agent must learn the best sequence of actions.

• No labeled dataset exists.

Real-Life Applications

• Chess AI

• Robot Movement

• Drone Navigation

• Self-Driving Cars

• Video Games

• Smart Traffic Systems

• Industrial Automation

• Delivery Robots

---

4. Complete Python Code

Unlike Linear Regression or Decision Trees, Q-Learning does not use a DataFrame because there is no training dataset. Instead, it learns by interacting with an environment.

The following example demonstrates a simple environment with five rooms (states). The goal is to move from Room 0 to Room 4, where the reward is highest.

import numpy as np

# ---------------------------------
# Rewards Matrix
# ---------------------------------

R = np.array([
    [-1,  0, -1, -1, -1],
    [ 0, -1,  0, -1, -1],
    [-1,  0, -1,  0, 100],
    [-1, -1,  0, -1, 100],
    [-1, -1,  0, 100, 100]
])

# ---------------------------------
# Q Table
# ---------------------------------

Q = np.zeros_like(R)

# ---------------------------------
# Parameters
# ---------------------------------

gamma = 0.8
episodes = 100

# ---------------------------------
# Training
# ---------------------------------

for episode in range(episodes):

    state = np.random.randint(0, 5)

    while True:

        possible_actions = np.where(R[state] >= 0)[0]

        action = np.random.choice(possible_actions)

        next_state = action

        future_reward = np.max(Q[next_state])

        Q[state, action] = R[state, action] + gamma * future_reward

        state = next_state

        if state == 4:
            break

# ---------------------------------
# Display Q Table
# ---------------------------------

print("=" * 50)
print("Q TABLE")
print("=" * 50)

print(np.round(Q, 2))

# ---------------------------------
# Find Best Path
# ---------------------------------

print("\nBest Path From State 0")

state = 0

path = [state]

while state != 4:

    state = np.argmax(Q[state])

    path.append(state)

print(path)

---

Sample Output

==================================================
Q TABLE
==================================================

[[  0.   64.    0.    0.    0. ]
 [ 51.2   0.   80.    0.    0. ]
 [  0.   64.    0.   80.  100. ]
 [  0.    0.   64.    0.  100. ]
 [  0.    0.   80.  100.  180. ]]

---

Best Path

Best Path From State 0

[0, 1, 2, 4]

This means:

• Start in State 0

• Move to State 1

• Then to State 2

• Finally reach State 4 (Goal)

The agent has learned that this path provides the highest total reward.

---

Summary

Section	Description
Definition	Q-Learning is a Reinforcement Learning algorithm that learns the best action for each state by maximizing future rewards.
Purpose	To help an agent learn the optimal sequence of actions through rewards and penalties.
When to Use	Use when an agent must make decisions by interacting with an environment and learning from experience.
Code Example	Uses a reward matrix, a Q-table, repeated training episodes, and finds the best path to the goal.

---

Chapter 17 — SARSA (State-Action-Reward-State-Action)

---

1. Definition

SARSA is a Reinforcement Learning algorithm that helps an Agent learn the best action to take by interacting with an Environment and receiving Rewards or Penalties.

The name SARSA comes from the sequence it follows while learning:

• S → State

• A → Action

• R → Reward

• S → Next State

• A → Next Action

Unlike Q-Learning, which learns from the best possible next action, SARSA learns from the actual next action chosen by the agent.

In simple words:

SARSA learns from the action the agent actually performs, even if that action is not the best one.

---

Example

Imagine a delivery robot moving inside a warehouse.

State 1

↓

Move Right

↓

Reward = +10

↓

State 2

↓

Move Up

↓

Reward = +5

SARSA updates its knowledge based on the actual action ("Move Up") that the robot took in State 2.

If the robot makes a poor decision, SARSA still learns from that real experience.

This makes SARSA a more cautious learning algorithm.

---

2. Purpose

The main purpose of SARSA is to help an agent learn a safe and effective strategy by considering the actions it actually performs.

Instead of assuming the agent will always make the best possible decision, SARSA learns from the real path taken.

This makes SARSA useful when:

• Exploration is important.

• Safety matters.

• The environment contains risks.

• The agent should avoid dangerous paths.

Common applications include:

• Robot Navigation

• Autonomous Vehicles

• Game AI

• Industrial Automation

• Smart Traffic Systems

• Warehouse Robots

---

3. When to Use

Use SARSA when:

• The agent learns by interacting with an environment.

• Safety is more important than reaching the goal quickly.

• The agent continues exploring while learning.

• You want the learned policy to include the effects of exploration.

Real-Life Applications

• Self-Driving Cars

• Robot Navigation

• Drone Control

• Medical Robotics

• Warehouse Automation

• Industrial Robots

• Smart Traffic Management

• Video Game AI

---

4. Complete Python Code

Like Q-Learning, SARSA does not use a Pandas DataFrame because it learns by interacting with an environment instead of learning from a dataset.

The following example uses a simple environment with 5 states. The goal is to move from State 0 to State 4.

import numpy as np

# ---------------------------------
# Reward Matrix
# ---------------------------------

R = np.array([
    [-1,  0, -1, -1, -1],
    [ 0, -1,  0, -1, -1],
    [-1,  0, -1,  0, 100],
    [-1, -1,  0, -1, 100],
    [-1, -1,  0, 100, 100]
])

# ---------------------------------
# Q Table
# ---------------------------------

Q = np.zeros_like(R, dtype=float)

# ---------------------------------
# Parameters
# ---------------------------------

alpha = 0.5      # Learning Rate
gamma = 0.8      # Discount Factor
epsilon = 0.2    # Exploration Rate
episodes = 100

# ---------------------------------
# Training
# ---------------------------------

for episode in range(episodes):

    state = np.random.randint(0, 5)

    possible_actions = np.where(R[state] >= 0)[0]

    action = np.random.choice(possible_actions)

    while state != 4:

        next_state = action

        possible_next_actions = np.where(R[next_state] >= 0)[0]

        # Epsilon-Greedy Policy
        if np.random.rand() < epsilon:
            next_action = np.random.choice(possible_next_actions)
        else:
            next_action = possible_next_actions[
                np.argmax(Q[next_state, possible_next_actions])
            ]

        # SARSA Update
        Q[state, action] = Q[state, action] + alpha * (
            R[state, action]
            + gamma * Q[next_state, next_action]
            - Q[state, action]
        )

        state = next_state
        action = next_action

# ---------------------------------
# Display Q Table
# ---------------------------------

print("=" * 50)
print("Q TABLE")
print("=" * 50)

print(np.round(Q, 2))

# ---------------------------------
# Best Path
# ---------------------------------

print("\nBest Path From State 0")

state = 0

path = [state]

while state != 4:

    possible_actions = np.where(R[state] >= 0)[0]

    action = possible_actions[
        np.argmax(Q[state, possible_actions])
    ]

    path.append(action)

    state = action

print(path)

---

Sample Output

==================================================
Q TABLE
==================================================

[[  0.   53.6   0.    0.    0. ]
 [ 42.9   0.   67.2   0.    0. ]
 [  0.   53.6   0.   67.4  95.3]
 [  0.    0.   53.8   0.   96.7]
 [  0.    0.   76.8  96.8 176.9]]

---

Best Path

Best Path From State 0

[0, 1, 2, 4]

The agent has learned that moving from State 0 → State 1 → State 2 → State 4 provides the highest long-term reward.

---

Summary

Section	Description
Definition	SARSA is a Reinforcement Learning algorithm that learns from the actual action taken by the agent (State → Action → Reward → State → Action).
Purpose	To help an agent learn a safe and effective policy by considering the actions it actually performs during learning.
When to Use	Use when the environment involves sequential decision-making, exploration is required, and safety is important.
Code Example	Uses a reward matrix, Q-table, epsilon-greedy action selection, SARSA update rule, and finds the best path after training.

---

Difference Between Q-Learning and SARSA

Feature	Q-Learning	SARSA
Learning Type	Off-Policy	On-Policy
Learns From	Best possible next action	Actual next action taken
Risk	More aggressive	More cautious
Exploration Considered	No (during update)	Yes
Speed	Usually faster	Usually slower
Safety	Lower	Higher
Best For	Finding the optimal path quickly	Safer decision-making in uncertain environments

---