How to Split a Dataset: Train, Validation, and Test Sets Explained | Day 22

Imagine a student who memorizes every single practice question and answer key before a final exam. On exam day, if the test features the exact same questions from the practice book, the student gets a perfect 100%. But if the test contains brand-new questions, the student fails completely.

Did the student actually learn the subject, or did they just memorize the practice sheet?

In Day 21: What is a Dataset?, we learned that a dataset is a structured collection of observations—the raw material AI systems use to learn patterns. But having data is only half the battle. If you feed all of your data to a machine learning algorithm during training, you will never know whether it truly learned general principles or simply memorized the answers.

Today, we will solve this fundamental challenge by learning how to split a dataset correctly into training, validation, and test subsets.


The Mental Model: Textbook, Practice Test, and Final Exam

The 3-Part Exam Analogy

To evaluate whether a machine learning model can perform on unseen real-world data, we divide our dataset into three separate portions:

  • Training Set (The Textbook): The material the model reads and learns from repeatedly to uncover patterns.
  • Validation Set (The Practice Quiz): The mid-study check used to tweak study habits, tune model settings (hyperparameters), and pick the best approach.
  • Test Set (The Final Exam): The strict, untouched evaluation used only at the very end to measure true performance on brand-new material.

Where the analogy breaks down: A human student can reflect on where they made mistakes during a final exam and try to study differently next semester. In machine learning, once you evaluate your model on the test set, you cannot go back and tweak your model based on those final exam results. If you do, your test set is no longer an unbiased metric!


The Three Core Subsets Explained

Let me break down why each of these three splits exists and how they function together in the machine learning workflow.

1. The Training Set

This is the largest portion of your data. The machine learning algorithm processes these examples to compute weights, detect trends, and establish decision boundaries. If you are predicting house prices, the training set contains the historical house characteristics and their actual sale prices.

2. The Validation Set

While the model is training, we need a way to check if it is generalizing well without showing it the final exam questions. The validation set helps us fine-tune parameters (known as hyperparameters, like learning rates or decision tree depth) and decide when to stop training before the model starts memorizing noise.

3. The Test Set

The test set serves as a realistic simulation of the future. It remains locked away in a vault until training and parameter tuning are completely finished. Its sole purpose is to provide an objective score of how well your model will perform when deployed in the real world.

Standard Ratio Guidelines:

  • Traditional ML (Small/Medium Data): 70% Train / 15% Validation / 15% Test (or 80% Train / 10% Validation / 10% Test).
  • Modern Deep Learning (Millions of rows): 98% Train / 1% Validation / 1% Test. When you have 10 million rows, 1% still provides 100,000 samples for evaluation, which is plenty for evaluation!

Technical Example: Splitting Data in Python

Let’s look at how data scientists split a dataset using Python and the popular scikit-learn library. We perform a two-step split to get our train, validation, and test sets.

# Importing the splitting utility from scikit-learn
from sklearn.model_selection import train_test_split
import pandas as pd

# Assume 'X' represents our feature matrix and 'y' represents target labels
# Step 1: Split total dataset into Train (80%) and Temporary Test/Val (20%)
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, 
    test_size=0.20, 
    random_state=42  # Ensures reproducible results across runs
)

# Step 2: Split the 20% temporary set equally into Validation (10%) and Test (10%)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, 
    test_size=0.50, 
    random_state=42
)

# Output summary of dataset proportions
print(f"Training samples: {len(X_train)}")
print(f"Validation samples: {len(X_val)}")
print(f"Test samples: {len(X_test)}")

Two Critical Pitfalls to Avoid

1. Data Leakage (The Cardinal Sin)

Data leakage occurs when information from outside the training dataset (such as validation or test data) unintentionally leaks into the model during training. For instance, if you calculate the overall average age of all users across your entire dataset before splitting, the training set now contains information about the test set. Always compute transformations (like mean scaling or normalization) only on the training split first!

2. Class Imbalance & Stratification

Imagine you are building a fraud detection model where 99% of transactions are legitimate and 1% are fraudulent. If you split your data completely at random, your validation or test set might end up with zero fraudulent transactions!

To prevent this, we use Stratified Splitting. Stratification ensures that each split maintains the exact same percentage of target categories as the full dataset.


Summary Comparison of Dataset Splits

Split Name Primary Goal Used By Analogy
Training Set Learn parameters & weight patterns The learning algorithm Course Textbook
Validation Set Tune hyperparameters & prevent overfitting The Data Scientist / Engineer Practice Exam
Test Set Provide final, unbiased quality score Project Stakeholders / Users Final Exam

Coach’s Note

Remember this golden rule: Treat your test set like a sealed envelope. Open it only once when you are convinced your model is completely ready. If you use test results to keep tweaking your model parameters day after day, your test set slowly becomes just another training set!



Quick Recall Checkpoint

Test your understanding before moving forward!

Question 1: Why shouldn’t we use the test set to tune model hyperparameters?

Because if you use the test set to tune hyperparameters, the model is indirectly learning to optimize for the test set. It loses its ability to serve as an independent, unbiased evaluation of unseen real-world data.

Question 2: What is “Stratified Splitting” and when should you use it?

Stratified splitting ensures that each split (train, validation, test) retains the exact same percentage of target classes as the complete dataset. You should use it whenever working with imbalanced datasets (e.g., medical diagnosis or fraud detection).

Question 3: What parameter in Python ensures that your split is reproducible every time you run the script?

The random_state parameter (e.g., random_state=42) seeds the random number generator, ensuring identical splits across execution runs.


Key Takeaways

  • Splitting datasets prevents models from simply memorizing data instead of generalizing to real scenarios.
  • The Training Set is used to fit the model parameters.
  • The Validation Set is used to tune settings and choose the best model architecture.
  • The Test Set is used strictly at the end as an unbiased measure of accuracy.
  • Always split before preprocessing to prevent Data Leakage.

What Comes Next?

Now that you know how to split your dataset to measure performance accurately, what happens when a model performs amazingly well on the training set but fails miserably on the validation set?

Tomorrow, in Day 23: Overfitting Explained, we will explore the single most common disease in machine learning and learn how to diagnose and fix it!

Prefer learning in Hinglish? Check out the Hinglish version of 365 Days of AI.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top