Chapter 8 — Encoding in Machine Learning
1. What is Encoding?
Encoding is the process of converting text (categorical data) into numbers so that Machine Learning algorithms can understand it.
In simple words,
Encoding means converting words into numbers.
2. Why Do We Need Encoding?
Imagine you have the following data:
| Customer | Gender |
|---|---|
| A | Male |
| B | Female |
| C | Male |
| D | Female |
A human can easily understand:
Male
Female
But a Machine Learning algorithm cannot.
Algorithms perform mathematical calculations such as:
Addition
Multiplication
Distance calculation
Probability calculation
These operations require numbers, not text.
Therefore, we must convert text into numbers.
3. What is Categorical Data?
Categorical data contains labels or names instead of numerical values.
Examples:
| Gender |
|---|
| Male |
| Female |
| City |
|---|
| Delhi |
| Mumbai |
| Pune |
| Department |
|---|
| HR |
| Sales |
| IT |
These values represent categories rather than quantities.
4. Types of Categorical Data
There are two types of categorical data.
A. Ordinal Data (Ordered Categories)
Ordinal categories have a meaningful order.
Example:
| Size |
|---|
| Small |
| Medium |
| Large |
There is a natural ranking:
Small < Medium < Large
Another example:
| Education |
|---|
| High School |
| Graduate |
| Postgraduate |
| PhD |
These categories also have an order.
B. Nominal Data (Unordered Categories)
Nominal categories do not have any natural order.
Example:
| City |
|---|
| Delhi |
| Mumbai |
| Pune |
Is Delhi greater than Mumbai?
No.
Is Pune smaller than Delhi?
No.
Cities are simply names.
Another examples:
Gender
Blood Group
Country
Department
Color
These categories have no ranking.
5. Types of Encoding
The two most common encoding techniques are:
Label Encoding
One-Hot Encoding
6. Label Encoding
Label Encoding converts each category into a unique number.
Example:
Original Data
| Gender |
|---|
| Male |
| Female |
| Male |
| Female |
After Label Encoding
| Gender |
|---|
| 1 |
| 0 |
| 1 |
| 0 |
Python Code
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["Gender"] = le.fit_transform(df["Gender"])
Output:
Male → 1
Female → 0
(The exact numbers may vary depending on how the encoder assigns labels.)
7. Another Example of Label Encoding
Original Data
| Membership |
|---|
| Silver |
| Gold |
| Platinum |
Encoded Data
| Membership |
|---|
| 1 |
| 0 |
| 2 |
Each category receives a unique number.
8. Problem with Label Encoding
Suppose we encode cities.
Original Data
| City |
|---|
| Delhi |
| Mumbai |
| Pune |
After Label Encoding
| City |
|---|
| 0 |
| 1 |
| 2 |
The algorithm may assume:
Pune > Mumbai > Delhi
But cities have no ranking.
The assigned numbers create a false order that does not exist in reality.
This can reduce the performance of some algorithms.
9. One-Hot Encoding
One-Hot Encoding creates a new column for every category.
Original Data
| City |
|---|
| Delhi |
| Mumbai |
| Pune |
After One-Hot Encoding
| Delhi | Mumbai | Pune |
|---|---|---|
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
Each row contains:
1 → Category exists
0 → Category does not exist
10. Python Code for One-Hot Encoding
df = pd.get_dummies(
df,
columns=["City"]
)
Output
| City_Delhi | City_Mumbai | City_Pune |
|---|---|---|
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
11. Label Encoding vs One-Hot Encoding
Suppose the dataset contains:
| Fruit |
|---|
| Apple |
| Banana |
| Mango |
Label Encoding
| Fruit |
|---|
| 0 |
| 1 |
| 2 |
The model may think:
Apple < Banana < Mango
This ordering is artificial.
One-Hot Encoding
| Apple | Banana | Mango |
|---|---|---|
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
Now the model treats all fruits equally.
12. Advantages of Label Encoding
Easy to apply.
Uses only one column.
Faster processing.
Saves memory.
Good for ordered categories.
13. Disadvantages of Label Encoding
Can introduce a false order.
Not suitable for most nominal data.
May confuse linear models.
14. Advantages of One-Hot Encoding
No false ordering.
Suitable for unordered categories.
Works well with many Machine Learning algorithms.
Easy to interpret.
15. Disadvantages of One-Hot Encoding
If a column contains many categories, many new columns are created.
Example:
100 Cities
↓
100 New Columns
This increases memory usage and can slow down model training.
16. When Should You Use Label Encoding?
Use Label Encoding when the categories have a natural order.
Examples:
| Feature | Ordered? | Use Label Encoding? |
|---|---|---|
| Small, Medium, Large | Yes | ✅ Yes |
| Poor, Average, Good | Yes | ✅ Yes |
| Bronze, Silver, Gold | Yes | ✅ Yes |
| Beginner, Intermediate, Expert | Yes | ✅ Yes |
17. When Should You Use One-Hot Encoding?
Use One-Hot Encoding when the categories have no order.
Examples:
| Feature | Ordered? | Use One-Hot Encoding? |
|---|---|---|
| City | No | ✅ Yes |
| Country | No | ✅ Yes |
| Gender | No | ✅ Yes |
| Department | No | ✅ Yes |
| Blood Group | No | ✅ Yes |
| Color | No | ✅ Yes |
18. Which Algorithms Prefer Which Encoding?
| Algorithm | Label Encoding | One-Hot Encoding | Reason |
|---|---|---|---|
| Linear Regression | ❌ Usually No | ✅ Yes | Linear models may misinterpret arbitrary numeric labels. |
| Logistic Regression | ❌ Usually No | ✅ Yes | One-hot avoids introducing false order. |
| Decision Tree | ✅ Yes | ✅ Yes | Trees split on values and are less affected by arbitrary labels. |
| Random Forest | ✅ Yes | ✅ Yes | Same reason as Decision Trees. |
| XGBoost | ✅ Yes | ✅ Yes | Tree-based algorithm. |
| K-Nearest Neighbors (KNN) | ❌ Usually No | ✅ Yes | Distance calculations are affected by arbitrary numeric labels. |
| Support Vector Machine (SVM) | ❌ Usually No | ✅ Yes | Works better without artificial ordering. |
19. Real-Life Examples
Example 1 – Student Grades
| Grade |
|---|
| Poor |
| Average |
| Good |
| Excellent |
Since there is an order:
Poor < Average < Good < Excellent
Use Label Encoding.
Example 2 – Customer City
| City |
|---|
| Delhi |
| Mumbai |
| Pune |
Cities have no ranking.
Use One-Hot Encoding.
Example 3 – Product Size
| Size |
|---|
| Small |
| Medium |
| Large |
There is a clear order.
Use Label Encoding.
Example 4 – Department
| Department |
|---|
| HR |
| Sales |
| IT |
No department is "greater" than another.
Use One-Hot Encoding.
20. Common Interview Questions
Q1. What is Encoding?
Answer:
Encoding is the process of converting categorical (text) data into numerical values so that Machine Learning algorithms can process it.
Q2. Why is Encoding Required?
Answer:
Machine Learning algorithms perform mathematical operations and cannot directly understand text values.
Q3. What is Label Encoding?
Answer:
Label Encoding converts each category into a unique numeric value. It is best suited for ordered (ordinal) categories.
Q4. What is One-Hot Encoding?
Answer:
One-Hot Encoding creates a separate binary (0/1) column for each category. It is best suited for unordered (nominal) categories.
Q5. When Should You Use Label Encoding?
Answer:
Use it for ordinal data where the categories have a meaningful order, such as Small, Medium, and Large.
Q6. When Should You Use One-Hot Encoding?
Answer:
Use it for nominal data where the categories have no natural order, such as City, Gender, or Department.
21. Chapter Summary
| Situation | Recommended Encoding |
|---|---|
| Ordered categories (Ordinal) | Label Encoding |
| Unordered categories (Nominal) | One-Hot Encoding |
| Few unordered categories | One-Hot Encoding |
| Many ordered categories | Label Encoding |
| Linear Regression | One-Hot Encoding |
| Logistic Regression | One-Hot Encoding |
| Decision Tree | Label Encoding or One-Hot Encoding |
| Random Forest | Label Encoding or One-Hot Encoding |
| XGBoost | Label Encoding or One-Hot Encoding |
Step 1: Is there a natural order?
│
Yes ──┴── No
│ │
Label Encoding One-Hot Encoding
Remember this simple rule:
Ordered categories → Label Encoding
Unordered categories → One-Hot Encoding
Example 1: Logistic Regression with Label Encoding
Problem
Predict whether an employee will be promoted.
Since Education has a natural order:
High School
Graduate
Postgraduate
PhD
we use Label Encoding.
Step 1: Create Dataset
import pandas as pd
df = pd.DataFrame({
"Education": [
"High School",
"Graduate",
"Postgraduate",
"PhD",
"Graduate",
"High School",
"Postgraduate",
"PhD"
],
"Experience":[
2,4,6,8,5,1,7,9
],
"Promoted":[
"No","No","Yes","Yes",
"Yes","No","Yes","Yes"
]
})
print(df)
Output
| Education | Experience | Promoted |
|---|---|---|
| High School | 2 | No |
| Graduate | 4 | No |
| Postgraduate | 6 | Yes |
| PhD | 8 | Yes |
| Graduate | 5 | Yes |
| High School | 1 | No |
| Postgraduate | 7 | Yes |
| PhD | 9 | Yes |
Step 2: Label Encoding
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["Education"] = le.fit_transform(df["Education"])
df["Promoted"] = le.fit_transform(df["Promoted"])
print(df)
Example Output
| Education | Experience | Promoted |
|---|---|---|
| 1 | 2 | 0 |
| 0 | 4 | 0 |
| 2 | 6 | 1 |
| 3 | 8 | 1 |
| 0 | 5 | 1 |
| 1 | 1 | 0 |
| 2 | 7 | 1 |
| 3 | 9 | 1 |
Step 3: Split Features and Target
X = df[["Education","Experience"]]
y = df["Promoted"]
Step 4: Train-Test Split
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
Step 5: Train Logistic Regression
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(X_train,y_train)
Step 6: Prediction
prediction = model.predict(X_test)
print(prediction)
Step 7: Predict New Employee
new_employee = [[2,7]]
result = model.predict(new_employee)
print(result)
Output
[1]
Meaning
Promoted = Yes
Example 2: Logistic Regression with One-Hot Encoding
Problem
Predict whether a customer will purchase a product.
City has no natural order.
So we use One-Hot Encoding.
Step 1: Create Dataset
import pandas as pd
df = pd.DataFrame({
"City":[
"Delhi",
"Mumbai",
"Pune",
"Delhi",
"Bangalore",
"Mumbai",
"Pune",
"Delhi"
],
"Income":[
30000,
60000,
45000,
28000,
70000,
65000,
50000,
32000
],
"Purchased":[
"No",
"Yes",
"Yes",
"No",
"Yes",
"Yes",
"Yes",
"No"
]
})
print(df)
Step 2: One-Hot Encoding
df = pd.get_dummies(
df,
columns=["City"]
)
Step 3: Encode Target
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df["Purchased"] = le.fit_transform(
df["Purchased"]
)
Step 4: Features and Target
X = df.drop(
"Purchased",
axis=1
)
y = df["Purchased"]
Step 5: Train-Test Split
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(
X,
y,
test_size=0.2,
random_state=42
)
Step 6: Train Logistic Regression
from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(
X_train,
y_train
)
Step 7: Prediction
prediction = model.predict(X_test)
print(prediction)
Step 8: Predict New Customer
Suppose the customer is from Pune with an income of ₹55,000.
The encoded features are:
| Income | City_Bangalore | City_Delhi | City_Mumbai | City_Pune |
|---|---|---|---|---|
| 55000 | 0 | 0 | 0 | 1 |
new_customer = [[
55000,
0,
0,
0,
1
]]
result = model.predict(new_customer)
print(result)
Output
[1]
Meaning
Purchased = Yes
| Label Encoding Example | One-Hot Encoding Example |
|---|---|
| Feature: Education | Feature: City |
| Categories have an order | Categories have no order |
| High School → Graduate → Postgraduate → PhD | Delhi, Mumbai, Pune, Bangalore |
Use LabelEncoder() | Use pd.get_dummies() |
| Target: Promoted | Target: Purchased |
| Algorithm: Logistic Regression | Algorithm: Logistic Regression |
Rule to Remember
Ordered categories (ordinal) →
LabelEncoder()Unordered categories (nominal) →
pd.get_dummies()orOneHotEncoder()
0 Comments