Decision Tree
1. What is a Decision Tree?
A Decision Tree is a Machine Learning algorithm that makes decisions using questions.
Just like humans.
Example:
Is Age > 25?
Yes → Next Question
No → Reject
Another example:
CartItems > 2 ?
Yes → Purchase
No → No Purchase
It looks like a tree.
CartItems > 2?
/ \
No Yes
| |
No Purchase Purchase
2. Why is it called a Tree?
Because it has:
Root Node
Branch
Leaf Node
Example:
CartItems > 2
/ \
No Yes
| |
No Buy Visits > 5
/ \
No Yes
| |
No Buy Buy
Looks like an upside-down tree.
3. Important Terminology
Root Node
First question.
Example:
CartItems > 2 ?
Root Node = Starting point.
Branch
Path connecting nodes.
Example:
Yes
No
Internal Node
Questions inside the tree.
Example:
WebsiteVisits > 5 ?
Leaf Node
Final answer.
Example:
Purchase
No Purchase
No more questions.
4. How Decision Tree Learns
Suppose data:
| CartItems | Purchased |
|---|---|
| 0 | No |
| 1 | No |
| 2 | Yes |
| 3 | Yes |
| 4 | Yes |
Tree sees:
CartItems > 1 ?
This perfectly separates data.
So it creates:
CartItems > 1 ?
as root node.
5. What is Classification?
Predict categories.
Example:
Yes / No
Pass / Fail
Spam / Not Spam
Your dataset:
Purchased
Yes
No
This is Classification.
Decision Tree Classifier is used.
from sklearn.tree import DecisionTreeClassifier
6. Load Dataset
import pandas as pd
df = pd.read_csv("data.csv")
7. Understand Data
df.head()
Shows first rows.
df.info()
Shows:
Column names
Data types
Null values
df.describe()
Shows:
Mean
Max
Min
Count
8. Data Cleaning
Very important.
Real data contains:
Missing values
Duplicate values
Wrong values
Check Missing Values
df.isnull().sum()
Example:
Gender 2
Income 1
Fill Missing Numerical Values
df["Income"] = df["Income"].fillna(
df["Income"].mean()
)
Fill Missing Text Values
df["Gender"] = df["Gender"].fillna(
df["Gender"].mode()[0]
)
Remove Duplicates
df.drop_duplicates(inplace=True)
9. Why Encoding?
Machine Learning understands:
Numbers
Not:
Male
Female
Delhi
Mumbai
Need conversion.
10. Label Encoding
Convert text to numbers.
Example:
Male = 0
Female = 1
Code:
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["Gender"] = le.fit_transform(
df["Gender"]
)
11. One Hot Encoding
For multiple categories.
Example:
Delhi
Mumbai
Pune
Becomes:
| Delhi | Mumbai | Pune |
|---|---|---|
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
Code:
df = pd.get_dummies(
df,
columns=["City"]
)
12. Separate X and y
X = Features
y = Target
Features:
X = df.drop(
"Purchased",
axis=1
)
Target:
y = df["Purchased"]
13. Train Test Split
Why?
Need unseen data.
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
14. Create Decision Tree
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier()
15. Train Model
model.fit(
X_train,
y_train
)
Model learns patterns.
16. Prediction
y_pred = model.predict(X_test)
17. Accuracy
Most common metric.
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(
y_test,
y_pred
)
Formula:
Correct Predictions
-------------------
Total Predictions
Example:
95/100
95%
18. Confusion Matrix
Most important interview topic.
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(
y_test,
y_pred
)
Example:
Predicted
No Yes
Actual No 8 1
Actual Yes 2 9
TP
Predicted Yes
Actually Yes
Correct
TN
Predicted No
Actually No
Correct
FP
Predicted Yes
Actually No
Wrong
FN
Predicted No
Actually Yes
Wrong
19. Precision
from sklearn.metrics import precision_score
precision = precision_score(
y_test,
y_pred
)
Meaning:
When model predicts YES,
how often is it correct?
20. Recall
from sklearn.metrics import recall_score
recall = recall_score(
y_test,
y_pred
)
Meaning:
How many actual YES cases
did model find?
21. F1 Score
Combination of:
Precision
Recall
from sklearn.metrics import f1_score
f1 = f1_score(
y_test,
y_pred
)
22. Feature Importance
Decision Tree's superpower.
for feature, importance in zip(
X.columns,
model.feature_importances_
):
print(feature, importance)
Example:
CartItems 0.60
Income 0.20
Visits 0.10
Meaning:
CartItems affects prediction most.
23. Visualize Tree
from sklearn.tree import plot_tree
import matplotlib.pyplot as plt
plt.figure(figsize=(15,10))
plot_tree(
model,
feature_names=X.columns,
filled=True
)
plt.show()
24. What is Entropy?
Measures disorder.
Example:
5 Yes
5 No
Very mixed.
Entropy = High
Example:
10 Yes
0 No
Pure.
Entropy = Low
Goal:
Reduce Entropy
25. What is Gini Index?
Another impurity measure.
Used by default.
DecisionTreeClassifier(
criterion="gini"
)
26. Information Gain
Formula:
Information Gain
=
Old Entropy
-
New Entropy
Highest gain wins.
27. How Root Node is Chosen
Decision Tree checks:
Age
Income
Visits
CartItems
Calculates:
Information Gain
Highest gain becomes:
Root Node
28. Overfitting
Dangerous topic.
Tree memorizes training data.
Example:
Training Accuracy = 100%
Testing Accuracy = 60%
Overfitting.
29. Underfitting
Tree too simple.
Example:
Training Accuracy = 50%
Testing Accuracy = 45%
Underfitting.
30. Max Depth
Controls tree size.
DecisionTreeClassifier(
max_depth=2
)
Small tree.
DecisionTreeClassifier(
max_depth=5
)
Medium tree.
DecisionTreeClassifier()
Unlimited depth.
31. Pruning
Cut unnecessary branches.
Purpose:
Reduce Overfitting
32. New Customer Prediction
new_customer = [[
30,
50000,
7,
20,
3
]]
prediction = model.predict(
new_customer
)
print(prediction)
Interview Questions You Must Answer
What is a Decision Tree?
A tree-based supervised learning algorithm that predicts outcomes using a series of decision rules.
What is Root Node?
The first decision/question in the tree.
What is Leaf Node?
The final prediction node.
What is Entropy?
Measure of disorder in data.
What is Gini Index?
Measure of impurity used for splitting nodes.
What is Information Gain?
Reduction in entropy after a split.
What is Overfitting?
Model memorizes training data and performs poorly on new data.
What is Underfitting?
Model is too simple and cannot learn patterns.
Why Encoding?
Machine Learning works with numbers, not text.
Does Decision Tree require scaling?
No.
Can Decision Tree handle non-linear data?
Yes.
Biggest advantage?
Easy to understand and visualize.
Biggest disadvantage?
Can overfit easily.
Decision Tree vs Logistic Regression
This is one of the most common interview questions.
Real-Life Analogy
Suppose you want to decide whether a customer will purchase.
Logistic Regression thinks like a mathematician
It says:
Let me calculate a score using a formula.
Example:
Score =
(Age × 0.2)
+
(Income × 0.0001)
+
(CartItems × 2)
Then it converts the score into a probability.
Purchase Probability = 85%
Prediction:
Yes
Decision Tree thinks like a human
It says:
CartItems > 2 ?
If Yes:
Purchase
If No:
WebsiteVisits > 5 ?
If Yes:
Purchase
Else:
No Purchase
How They Work
Logistic Regression
Uses a mathematical equation.
Features
↓
Equation
↓
Probability
↓
Prediction
Example:
Probability = 0.92
Prediction = Yes
Decision Tree
Uses questions.
Features
↓
Question 1
↓
Question 2
↓
Question 3
↓
Prediction
Example:
CartItems > 2 ?
Visualization
Logistic Regression
Age
Income
Visits
CartItems
↓
Mathematical Formula
↓
Probability
↓
Yes / No
Decision Tree
CartItems > 2
/ \
No Yes
| |
No Purchase Purchase
Output
Logistic Regression
Gives probability.
Example:
model.predict_proba(X)
Output:
[0.15 0.85]
Meaning:
15% No Purchase
85% Purchase
Decision Tree
Directly gives decision.
Purchase
or
No Purchase
Relationship with Data
Logistic Regression
Best when relationship is linear.
Example:
More Income
→ More Purchases
More Visits
→ More Purchases
Straightforward pattern.
Decision Tree
Handles complex patterns.
Example:
If Age > 25
AND
Income > 40000
AND
Visits > 5
THEN Purchase
Complex conditions.
Encoding
Both require encoding.
Example:
Male
Female
Delhi
Mumbai
Must become numbers.
Scaling
Logistic Regression
Scaling recommended.
StandardScaler()
because:
Income = 50000
Visits = 5
Income dominates.
Decision Tree
No scaling needed.
Works directly.
Huge advantage.
Feature Importance
Logistic Regression
Uses coefficients.
Example:
Age 0.2
Income 0.0001
CartItems 2.5
Interpretation:
CartItems most important
Decision Tree
Uses feature importance.
Example:
CartItems 0.60
Income 0.25
Visits 0.10
Interpretation:
CartItems most important
Overfitting
Logistic Regression
Less likely.
Because:
Simple equation.
Decision Tree
More likely.
Because:
Keeps creating branches.
Example:
Training Accuracy = 100%
Testing Accuracy = 60%
Overfitting.
Underfitting
Logistic Regression
Can underfit if data is complex.
Decision Tree
Can underfit if:
max_depth = 1
Tree too small.
Accuracy
Depends on dataset.
Linear dataset
Logistic Regression wins
Non-linear dataset
Decision Tree wins
Interpretability
Logistic Regression
Explanation:
Income contributes +0.5
Age contributes +0.2
Needs mathematics.
Decision Tree
Explanation:
CartItems > 2
→ Purchase
Very easy.
In Your Customer Dataset
Features:
Age
Income
Visits
TimeSpent
CartItems
Target:
Purchased
Logistic Regression learns
More CartItems
→ Higher purchase probability
More Visits
→ Higher purchase probability
Decision Tree learns
CartItems > 2 ?
Then:
Visits > 5 ?
Then:
Purchase
Interview Comparison Table
| Feature | Logistic Regression | Decision Tree |
|---|---|---|
| Algorithm Type | Statistical | Tree-Based |
| Uses Formula | Yes | No |
| Uses Questions | No | Yes |
| Output | Probability | Decision Rules |
| Scaling Required | Usually Yes | No |
| Handles Non-linear Data | Poorly | Very Well |
| Easy to Visualize | No | Yes |
| Feature Importance | Coefficients | Feature Importance |
| Overfitting Risk | Low | High |
| Speed | Faster | Fast |
| Missing Values | Handle First | Handle First |
| Best For | Linear Patterns | Complex Patterns |
Interview Answer (2 Marks)
What is the difference between Logistic Regression and Decision Tree?
Logistic Regression predicts outcomes using a mathematical equation and probability, while Decision Tree predicts outcomes using a sequence of if-else conditions. Logistic Regression works best for linear relationships and is less prone to overfitting, whereas Decision Trees can model complex non-linear relationships but may overfit if not properly controlled.
Logistic Regression
=
Formula + Probability
Decision Tree
=
Questions + Decisions
Let's see how a Decision Tree actually builds itself for each real-life example.
Example 1: Customer Purchase Prediction
Data
| Age | Income | Visits | CartItems | Purchased |
|---|---|---|---|---|
| 22 | 25000 | 2 | 0 | No |
| 30 | 50000 | 8 | 3 | Yes |
| 35 | 70000 | 12 | 5 | Yes |
| 25 | 30000 | 3 | 1 | No |
| 40 | 90000 | 15 | 6 | Yes |
Decision Tree
CartItems > 2 ?
/ \
No Yes
| |
No Purchase Purchase
Why?
Observe:
CartItems = 0,1
→ No
CartItems = 3,5,6
→ Yes
The tree finds this rule automatically.
Prediction
New customer:
Age = 28
Income = 45000
Visits = 7
CartItems = 4
Tree:
CartItems > 2 ?
4 > 2
Yes
Result:
Purchase = Yes
Example 2: Loan Approval
Data
| Income | CreditScore | Experience | LoanApproved |
|---|---|---|---|
| 25000 | 500 | 1 | No |
| 60000 | 750 | 5 | Yes |
| 80000 | 820 | 8 | Yes |
| 35000 | 550 | 2 | No |
| 100000 | 900 | 10 | Yes |
Decision Tree
CreditScore > 700 ?
/ \
No Yes
| |
Reject Approve
Why?
Notice:
500 → No
550 → No
750 → Yes
820 → Yes
900 → Yes
Credit score perfectly separates customers.
Prediction
New Applicant:
Income = 65000
CreditScore = 780
Experience = 6
Tree:
780 > 700
Yes
Result:
Loan Approved
Example 3: Employee Attrition
Data
| Salary | YearsAtCompany | LeftCompany |
|---|---|---|
| 25000 | 1 | Yes |
| 70000 | 8 | No |
| 90000 | 12 | No |
| 30000 | 2 | Yes |
| 100000 | 15 | No |
Decision Tree
YearsAtCompany > 5 ?
/ \
No Yes
| |
Leave Stay
Why?
Observe:
1 year → Leave
2 years → Leave
8 years → Stay
12 years → Stay
15 years → Stay
Tree discovers:
YearsAtCompany > 5
Prediction
Employee:
Salary = 50000
YearsAtCompany = 7
Tree:
7 > 5
Yes
Result:
Stay
How Decision Tree Thinks Internally
Suppose Customer Dataset:
| CartItems | Purchased |
|---|---|
| 0 | No |
| 1 | No |
| 2 | Yes |
| 3 | Yes |
| 4 | Yes |
Tree tests many questions:
CartItems > 0 ?
CartItems > 1 ?
CartItems > 2 ?
CartItems > 3 ?
Then calculates:
Entropy
or
Gini Index
and chooses the question that separates the data best.
Result:
CartItems > 1 ?
becomes Root Node.
Python Code
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
Visualizing the Tree
from sklearn.tree import plot_tree
import matplotlib.pyplot as plt
plt.figure(figsize=(12,8))
plot_tree(
model,
feature_names=X.columns,
class_names=["No","Yes"],
filled=True
)
plt.show()
Example output:
CartItems <= 2.5
├── True
│ └── No Purchase
└── False
└── Purchase
Interview Summary
Decision Tree always does:
Step 1:
Find best question
Step 2:
Split data
Step 3:
Find next best question
Step 4:
Repeat
Step 5:
Reach final answer
Example:
Customer Purchase
CartItems > 2 ?
Yes
↓
Purchase
No
↓
No Purchase
That is exactly how a Decision Tree solves real-world problems.
0 Comments