Training Data vs Testing Data: Machine Learning Mein Split Kyun Zaroori Hai? | Day 19

365 Days of AI • Day 19 | ML Fundamentals

Model ko ‘ratta’ marne se bachane aur real-world performance check karne ka sabse important concept.

Imagine kijiye ek student ki jo exam ki taiyari kar raha hai. Agar woh saare saal textbook ke same 10 solved questions ki practice kare aur exam mein bhi teacher wahi exact 10 questions pooch de, toh kya hoga? Student 100 out of 100 score kar lega!

Lekin kya iska matlab ye hai ki student ko subject sach mein samajh aaya hai? Bilkul nahi. Usne bas answers ‘ratta’ (memorize) kar liye the.

Machine Learning mein bhi exact yahi problem hoti hai. Agar hum kisi AI model ko jis data par train karte hain, usi data par uska test bhi le lein, toh model 100% accurate lag sakta hai—lekin real world mein severe fail ho sakta hai. Isiliye hum apne data ko do hisson mein baant te hain: Training Data aur Testing Data.


1. The Exam Mental Model (Mental Model)

Is concept ko humesha is simple mental model se yaad rakhiye:

  • Training Data = Classroom Practice & Solved Exercises: Yahan student (model) questions aur unke correct answers ko baar-baar dekh kar patterns samajhta hai.
  • Testing Data = Final Unseen Board Exam: Ye wo questions hain jo student ne pehle kabhi nahi dekhe. Yahan check hota hai ki student ne concept samjha hai ya sirf memorize kiya hai.

2. Training Data Kya Hai?

Training Data wo dataset hai jise hum Machine Learning algorithm ko pass karte hain taaki wo data ke andar majood underlying patterns, relationships, aur rules ko seekh sake.

Jab model Training Data par process hota hai, toh wo continuously apne internal parameters (weights and biases) ko adjust karta hai taaki errors kam se kam hon. Usually, poore dataset ka 70% se 80% hissa Training Data ke roop mein rakha jata hai.

Training Phase Mein Kya Hota Hai?

  • Model inputs (jaise ghar ka size, location) aur outputs (ghar ki keemat) dono dekhta hai.
  • Model in dono ke beech ka relationship figure out karta hai.
  • Model seekhta hai ki naye inputs aane par correct output kaise predict karna hai.

3. Testing Data Kya Hai?

Testing Data dataset ka wo chhota portion (normally 20% se 30%) hota hai jise training ke waqt model se poori tarah chhupa kar rakha jata hai. Is data ko Holdout Set ya Unseen Data bhi kehte hain.

Jab model ki training poori ho jaati hai, tab hum Testing Data ko model ke samne laate hain aur dekhte hain ki model kitne accurate predictions karta hai.

Golden Rule: Model training ke dauran Testing Data ko kabhi nahi dekhna chahiye! Agar model ne testing data pehle hi dekh liya, toh test score fake aur misleading ho jayega. Is mistake ko Data Engineering mein Data Leakage kehte hain.

4. Overfitting: Training vs Testing Split Kyun Zaroori Hai?

Machine Learning mein ek sabse badi problem hoti hai: Overfitting.

Overfitting tab hoti hai jab model Training Data ke patterns ke sath-sath uska ‘noise’ aur unwanted details bhi yaad kar leta hai. Iska result ye hota hai:

  • Training Data Par Accuracy: 99% (Perfect performance)
  • Testing Data Par Accuracy: 55% (Real-world failure)

Agar aapke paas Testing Data nahi hoga, toh aapko lagta rahega ki aapka model 99% accurate hai, aur jab aap use actual production mein launch karenge, toh wo fail ho jayega. Testing Data hume Overfitting pehchanne mein madad karta hai.

5. Training Data vs Testing Data: Quick Comparison

Feature Training Data Testing Data
Main Purpose Model ko patterns sikhana aur parameters train karna. Unseen data par model ka real performance evaluate karna.
Typical Size 70% – 80% dataset ka 20% – 30% dataset ka
Used During Model Building / Training Phase Evaluation / Testing Phase
Model Access Model is data aur iske labels ko directly dekhta hai. Model is data ko training ke waqt bilkul nahi dekhta.

6. Code Mein Train-Test Split Kaise Hota Hai?

Python aur Python ki famous Machine Learning library scikit-learn mein hum ye split train_test_split function ke zariye karte hain:

# Import the necessary utility from scikit-learn
from sklearn.model_selection import train_test_split

# Assume X contains input features (e.g., house size) and y contains target values (e.g., price)
# test_size=0.2 means 20% data for testing, 80% for training
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Total samples: {len(X)}")
print(f"Training samples: {len(X_train)}")  # 80%
print(f"Testing samples: {len(X_test)}")    # 20%

7. Beginners Ki Common Mistakes

  1. Poore Data Par Train Aur Test Karna: Sabse badi galti! Same data par train karke same data par test karne se accuracy 100% dikhai degi, jo ek illusion hai.
  2. Random Split Na Karna: Agar aapka data ordered hai (jaise pehle 100 rows sirf Apple phones ki hain aur aage ki 20 rows Samsung ki), aur aapne bina shuffle kiye split kar diya, toh model kabhi Samsung dekhega hi nahi!
  3. Testing Data Par Decisions Lena: Testing Data par model test karne ke baad agar aap baar-baar model ke settings change karte rehte hain testing score ko dekh kar, toh indirectly aapka model testing data se bias absorb kar leta hai.

Krish’s Coach Note 💡

“Jab bhi aap koi ML project start karein, sabse pehla kaam data ko split karna hota hai. Testing Data ko ek sealed envelope ki tarah samajhiye—ise tabhi kholna hai jab aapka model fully train ho chuka ho. Real-world AI engineering mein model ki success is baat se decide nahi hoti ki usne purana data kitna achha yaad rakha, balki is baat se hoti hai ki wo naye data par kitna smartly act karta hai.”

8. Mini Practical Thought Exercise 🧠

Maan lijiye aapke paas 1,000 spam aur non-spam emails ka dataset hai. Aap ek email filter model banana chahte hain.

  • Aap Training Set mein kitne emails rakhenge? (Hint: 80%)
  • Aap Testing Set mein kitne emails rakhenge? (Hint: 20%)
  • Agar aapka model Training Set par 98% score karta hai aur Testing Set par 60% score karta hai, toh aapka diagnosis kya hoga?

9. Quick Recall / Checkpoint 🎯

Apni understanding test karne ke liye niche diye gaye questions par click kijiye:

Q1: Training Data aur Testing Data ka standard split ratio kya hota hai?

Generally 80% Training Data aur 20% Testing Data rakha jata hai. Chhote datasets mein 70/30 ratio bhi common hai.

Q2: Overfitting kya hoti hai aur Testing Data ise kaise pakadta hai?

Overfitting tab hoti hai jab model training data par ratta mar leta hai. Testing data naye/unseen questions ki tarah kaam karta hai—agar model testing data par low accuracy deta hai, toh overfitting pakdi jaati hai.

Q3: Data Leakage kya hota hai?

Data Leakage tab hoti hai jab Testing Data ki information accidentally Training Data mein mix ho jaati hai, jisse test results unnaturally high dikhte hain.

10. Key Takeaways

  • Training Data model ko learn karne ke liye input parameters aur patterns deta hai.
  • Testing Data unseen data par model ki real generalization ability ko evaluate karta hai.
  • Training aur Testing data ko humesha alag (isolated) rakhna chahiye.
  • Training data par high score aur Testing data par low score Overfitting ka direct sign hai.

What Comes Next?

Aaj humne samjha ki model ko train aur test kaise kiya jata hai. Lekin is dataset ke andar actual columns mein kya hota hai?

Kal (Day 20) hum dekhenge: Features and Labels in Machine Learning. Hum samjhenge ki inputs (Features) aur outputs (Labels) kis tarah organize kiye jaate hain.

Poori learning journey dekhne ke liye visit karein: 365 Days of AI Hinglish Course Page.

English mein continue karna chahte hain? Visit: 365 Days of AI (English).

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top