1. Definition
Hyperparameter Tuning is the process of finding the best settings for a Machine Learning algorithm to improve its performance.
Every Machine Learning algorithm has some settings that control how it learns.
These settings are called Hyperparameters.
By changing these settings, we can:
Increase Accuracy
Reduce Errors
Prevent Overfitting
Improve Generalization
Example
Consider a Random Forest model.
RandomForestClassifier(
n_estimators=100,
max_depth=5
)
Here:
n_estimators = 100
max_depth = 5
are Hyperparameters.
They are chosen before training the model.
2. Purpose
The purpose of Hyperparameter Tuning is to find the best combination of settings that produces the best model performance.
Without tuning:
Accuracy = 75%
After tuning:
Accuracy = 90%
Same algorithm.
Same dataset.
Different settings.
Better result.
3. When to Use
Use Hyperparameter Tuning when:
The model accuracy is not satisfactory.
The model is overfitting.
The model is underfitting.
You want the best possible performance.
The project is production-ready.
Commonly Tuned Algorithms
Decision Tree
Random Forest
KNN
Logistic Regression
SVM
XGBoost
LightGBM
CatBoost
4. Parameter vs Hyperparameter
Many beginners confuse these terms.
Parameters
Parameters are learned automatically during training.
Example:
Linear Regression
y = mx + b
The model learns:
m (slope)
b (intercept)
These are Parameters.
Hyperparameters
Hyperparameters are chosen before training.
Example:
DecisionTreeClassifier(
max_depth=5
)
The value:
max_depth = 5
is a Hyperparameter.
Simple Comparison
| Parameters | Hyperparameters |
|---|---|
| Learned automatically | Set manually |
| Created during training | Set before training |
| Model learns them | User chooses them |
| Example: Coefficients | Example: max_depth |
Common Hyperparameters
Decision Tree
DecisionTreeClassifier(
max_depth=5,
min_samples_split=2
)
Important Hyperparameters
| Hyperparameter | Meaning |
|---|---|
| max_depth | Maximum tree depth |
| min_samples_split | Minimum records required to split |
| min_samples_leaf | Minimum records in a leaf node |
| criterion | Split method |
Random Forest
RandomForestClassifier(
n_estimators=100,
max_depth=10
)
| Hyperparameter | Meaning |
|---|---|
| n_estimators | Number of trees |
| max_depth | Tree depth |
| min_samples_split | Minimum split records |
| max_features | Features considered per split |
KNN
KNeighborsClassifier(
n_neighbors=5
)
| Hyperparameter | Meaning |
|---|---|
| n_neighbors | Number of neighbors |
| weights | Uniform or Distance |
| metric | Distance calculation |
What Happens Without Tuning?
Suppose:
Dataset = Product Sales
Model:
DecisionTreeClassifier()
Output:
Accuracy = 72%
After tuning:
DecisionTreeClassifier(
max_depth=5,
min_samples_leaf=3
)
Output:
Accuracy = 87%
The algorithm didn't change.
Only Hyperparameters changed.
Manual Hyperparameter Tuning
Example:
for k in range(1,11):
model = KNeighborsClassifier(
n_neighbors=k
)
model.fit(X_train,y_train)
score = model.score(
X_test,
y_test
)
print(k, score)
Output:
K=1 Accuracy=0.80
K=2 Accuracy=0.82
K=3 Accuracy=0.88
K=4 Accuracy=0.84
K=5 Accuracy=0.91
Best value:
K = 5
GridSearchCV
The most common tuning technique.
Instead of manually trying values:
max_depth = 3
max_depth = 4
max_depth = 5
max_depth = 6
GridSearchCV tries every combination automatically.
Example Using GridSearchCV
import pandas as pd
from sklearn.model_selection import (
train_test_split,
GridSearchCV
)
from sklearn.tree import (
DecisionTreeClassifier
)
# ---------------------------------
# Dataset
# ---------------------------------
data = {
"Age":[
22,25,28,30,35,
40,45,50,27,32,
24,29,36,42,48,
23,31,38,44,52
],
"Income":[
25000,30000,35000,42000,50000,
56000,62000,70000,32000,45000,
28000,38000,52000,60000,68000,
26000,43000,55000,63000,72000
],
"Purchased":[
0,0,0,0,1,
1,1,1,0,1,
0,0,1,1,1,
0,1,1,1,1
]
}
df = pd.DataFrame(data)
X = df[
[
"Age",
"Income"
]
]
y = df["Purchased"]
# ---------------------------------
# Split
# ---------------------------------
X_train, X_test, y_train, y_test = (
train_test_split(
X,
y,
test_size=0.30,
random_state=42
)
)
# ---------------------------------
# Model
# ---------------------------------
model = DecisionTreeClassifier()
# ---------------------------------
# Hyperparameters
# ---------------------------------
params = {
"max_depth":[
2,
3,
4,
5,
6
],
"criterion":[
"gini",
"entropy"
]
}
# ---------------------------------
# Grid Search
# ---------------------------------
grid = GridSearchCV(
estimator=model,
param_grid=params,
cv=5,
scoring="accuracy"
)
grid.fit(
X_train,
y_train
)
print("Best Parameters:")
print(grid.best_params_)
print("\nBest Score:")
print(grid.best_score_)
Example Output
Best Parameters:
{
'criterion': 'gini',
'max_depth': 3
}
Best Score:
0.92
Meaning:
GridSearchCV tested all combinations.
The best Decision Tree is:
max_depth = 3
criterion = gini
RandomizedSearchCV
Problem:
GridSearchCV tries every combination.
If there are many combinations:
10 × 10 × 10 × 10
=
10000 combinations
Training becomes slow.
Solution
Use:
RandomizedSearchCV
It tries only random combinations.
Much faster.
Example
from sklearn.model_selection import RandomizedSearchCV
random_search = RandomizedSearchCV(
estimator=model,
param_distributions=params,
n_iter=5,
cv=5
)
random_search.fit(
X_train,
y_train
)
print(
random_search.best_params_
)
GridSearchCV vs RandomizedSearchCV
| Feature | GridSearchCV | RandomizedSearchCV |
|---|---|---|
| Tests all combinations | Yes | No |
| Faster | No | Yes |
| More accurate search | Yes | Sometimes |
| Large datasets | Slow | Better |
| Small datasets | Excellent | Good |
Interview Questions
What is Hyperparameter Tuning?
Finding the best settings for a machine learning algorithm.
Why is Hyperparameter Tuning important?
It improves model performance and prevents overfitting or underfitting.
Difference between Parameter and Hyperparameter?
Parameters are learned by the model.
Hyperparameters are set before training.
Which methods are used for Hyperparameter Tuning?
Manual Search
GridSearchCV
RandomizedSearchCV
Which is faster?
RandomizedSearchCV
Summary
| Topic | Meaning |
|---|---|
| Hyperparameter | Setting chosen before training |
| Parameter | Value learned during training |
| GridSearchCV | Tests all combinations |
| RandomizedSearchCV | Tests random combinations |
| Goal | Find best model settings |
| Benefit | Better accuracy and performance |
0 Comments