Have you ever played the game 20 Questions? You start with a broad question—like “Is it living?”—and based on the answer, you ask progressively more specific questions until you arrive at the exact answer.
That simple, intuitive process is the foundational idea behind one of machine learning’s most practical and human-understandable algorithms: Decision Trees.
Welcome back to Day 33 of our 365 Days of AI journey! I’m Coach Krish, and today we’re stepping into the world of rule-based decision models.
Yesterday, on Day 32: Logistic Regression Explained, we looked at how machine learning calculates probabilities using linear boundaries and sigmoid curves. Today, we shift from drawing straight lines to building step-by-step logical flowcharts.
What You Will Learn Today
- The intuitive mental model behind Decision Trees
- Key anatomical parts of a tree: Root, Decision Nodes, and Leaves
- How a Decision Tree decides where to split (Gini Impurity and Information Gain explained simply)
- How to implement a Decision Tree in Python using
scikit-learn - The major vulnerability of Decision Trees: Overfitting
- How this concept sets the stage for our next lesson on Random Forests
The Mental Model: Machine Learning as a Flowchart
Mental Model: The Decision Flowchart
Imagine writing a rulebook for an everyday choice: Should I go for a outdoor run today?
You check the conditions step-by-step:
- Is it raining? If Yes → Stay inside. If No → Ask question 2.
- Is the humidity high? If Yes → Stay inside. If No → Go for a run!
A Decision Tree takes tabular data and automatically constructs this exact type of flowchart. Instead of a human manually writing the rules, the algorithm scans the data to discover which questions split the data most cleanly into final categories.
Anatomy of a Decision Tree
To understand how Decision Trees work, let me introduce three basic building blocks:
- Root Node: The top-level starting point containing the entire dataset. It asks the very first and most informative question.
- Decision Nodes (Internal Nodes): Subsequent branch points where the data is split again based on specific feature values.
- Leaf Nodes (Terminal Nodes): The final endpoints that contain predictions (e.g., “Classify as Spam” or “Predict Price = $250,000”). Leaf nodes do not split further.
How a Decision Tree Learns: The Splitting Mechanics
Imagine you have a bucket containing 50 red balls and 50 blue balls. The bucket is in a state of high chaos or high impurity.
If you can ask one question that separates all 50 red balls into Container A and all 50 blue balls into Container B, you have achieved perfect purity.
Decision trees use mathematical metrics to measure this purity and determine the best possible question to ask at each node:
1. Gini Impurity
Gini Impurity measures the probability that a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset.
- Gini = 0.0: Perfect purity (all items belong to a single class).
- Gini = 0.5: Maximum impurity in a binary system (50% Class A, 50% Class B).
2. Entropy and Information Gain
Entropy is a term borrowed from thermodynamics and information theory that measures randomness or uncertainty in data. When a split reduces entropy, we gain clear information. That reduction in uncertainty is called Information Gain.
The tree systematically tests every possible feature split and selects the split that yields the highest Information Gain (or lowest Gini Impurity).
Technical Example: Decision Tree in Python
Let’s see how simple it is to train a classification tree using Python’s scikit-learn framework:
# Importing the DecisionTreeClassifier from scikit-learn
from sklearn.tree import DecisionTreeClassifier
import numpy as np
# Sample Dataset: [Temperature (F), Humidity (%)]
X = np.array([
[85, 85], # Hot, Humid -> No Run (0)
[80, 90], # Hot, Humid -> No Run (0)
[70, 50], # Mild, Dry -> Run (1)
[65, 70], # Cool, Normal -> Run (1)
[72, 95] # Mild, Very Humid -> No Run (0)
])
# Labels: 1 = Run outdoors, 0 = Stay indoors
y = np.array([0, 0, 1, 1, 0])
# Initialize the Decision Tree Classifier
# We set max_depth to keep the tree simple and prevent overfitting
model = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=42)
# Train (fit) the model on our data
model.fit(X, y)
# Predict for a new day: Temperature = 75 F, Humidity = 55%
new_day = [[75, 55]]
prediction = model.predict(new_day)
print("Prediction (1 = Run, 0 = Stay):", prediction[0])
In just a few lines of code, the algorithm creates internal numerical thresholds (e.g., Humidity <= 80%) to predict outcomes for unseen data.
Strengths vs. Weaknesses
| Strengths | Weaknesses |
|---|---|
| Highly Explainable: Easy to visualize and explain to non-technical stakeholders. | Overfitting Risk: Deep trees memorize training noise easily (high variance). |
| No Feature Scaling Required: Doesn’t require normalization or standard scaling. | Instability: Small changes in data can lead to a completely different tree structure. |
| Handles Mixed Data: Works naturally with both numerical and categorical data. | Greedy Algorithm: Makes locally optimal choices at each node, which may not yield the global optimum. |
Common Misconceptions
Misconception: “Deeper trees are always more accurate.”
Reality: If you allow a Decision Tree to grow infinitely deep without stopping, it will create individual leaves for single data points. This creates a model that gets 100% accuracy on training data, but fails completely on real-world test data. This is classic overfitting.
max_depth, min_samples_split, or min_samples_leaf. Constraining the tree’s depth forces it to generalize rather than memorize.Mini Exercise: Thought Experiment
Imagine you are building a Decision Tree to classify whether an email is Spam or Not Spam.
You have three features available:
contains_word_winner(Yes / No)sender_in_contacts(Yes / No)email_length_words(Number)
Question: Which feature do you think would make the best Root Node split to immediately reduce the most uncertainty?
Quick Recall / Checkpoint
1. What is the top node of a decision tree called?
The top node is called the Root Node. It represents the initial state containing the entire dataset before any splits occur.
2. What does a Gini Impurity score of 0.0 mean?
A Gini Impurity score of 0.0 means the node is completely pure—every sample in that node belongs to exactly the same class.
3. Why do unconstrained decision trees overfit easily?
Because an unconstrained tree continues splitting until every single sample is isolated, essentially memorizing noisy details and specific training examples rather than discovering broad patterns.
Key Takeaways
- Decision Trees organize decisions into an intuitive flowchart of feature-based questions.
- Tree structure consists of a Root Node, Decision Nodes, and Leaf Nodes.
- Splits are chosen mathematically to maximize Information Gain or minimize Gini Impurity.
- Decision Trees are non-linear, interpretable, and require little data preparation, but are susceptible to high variance and overfitting.
- Setting
max_depthand other hyperparameters is crucial to building generalizable models.
What Comes Next?
Today, we learned that a single Decision Tree is easy to interpret, but can easily become unstable or overfit if left unchecked.
What if, instead of relying on a single tree, we grew an entire forest of trees and let them vote on the final answer?
Tomorrow, on Day 34, we explore Random Forest Explained—where we combine hundreds of decision trees to create one of the most reliable and powerful algorithms in machine learning.
Prefer learning in Hinglish? Continue with the Hinglish version of this course.