Yesterday, on Day 31: Linear Regression Explained, we mastered predicting continuous numeric values like house prices, test scores, or company revenue. But what happens when the answer isn’t a continuous number, but a discrete category? Is an incoming email spam or not spam? Will a bank transaction be fraudulent or legitimate? Will a patient test positive or negative?
Welcome to Day 32 of 365 Days of AI! I’m Coach Krish, and today we are stepping into the world of classification algorithms with one of the most foundational, battle-tested models in machine learning: Logistic Regression.
What You Will Learn Today
- Why standard linear regression fails when applied to yes/no classification tasks.
- How the Sigmoid Activation Function converts raw linear outputs into probability scores between 0 and 1.
- How a Decision Threshold translates probabilities into final categorical decisions.
- Why it is named “Logistic Regression” even though it is used for classification.
- How to implement a quick Logistic Regression model in Python using
scikit-learn.
1. The Problem: Why Linear Regression Fails for Classification
To understand why Logistic Regression exists, let’s first attempt a simple experiment: using standard Linear Regression for a binary decision (where class 0 = Not Spam and class 1 = Spam).
Linear Regression fits a straight line equation to our data points: y = m * x + b. However, this creates two major mathematical issues:
- Unbounded Outputs: A straight line stretches to positive infinity in one direction and negative infinity in the other. It can predict output values like
1.45or-0.32. What does a spam probability of-0.32actually mean? Probabilities must strictly exist between0.0(0%) and1.0(100%). - Sensitivity to Outliers: Adding just one extreme outlier (for instance, an email containing 1,000 spam keywords) tilts the slope of the entire regression line, severely distorting the boundary for normal emails.
2. The Solution: The Sigmoid Function
To fix this, we need a mathematical “wrapper” that takes any real value from negative infinity to positive infinity and squashes it smoothly into a range between 0 and 1.
Mental Model: The S-Curve Squeezer
Imagine the Sigmoid Function as an elastic S-shaped pipe. No matter how high or low a numerical score enters the pipe—whether it is +1,000 or -5,000—the pipe bends and squashes the output so it exits between 0.0 and 1.0.
Mathematically, the Sigmoid function $\sigma(z)$ is expressed as:
σ(z) = 1 / (1 + e^(-z))
Here, z represents the familiar linear equation output: z = (w1 * x1 + w2 * x2 + ... + b).
- When
zis extremely large and positive,e^(-z)approaches0, makingσ(z)approach1.0. - When
zis0,σ(z) = 1 / (1 + 1) = 0.5. - When
zis extremely large and negative,e^(-z)becomes massive, causingσ(z)to approach0.0.
3. From Probability to Decision: Setting the Threshold
Once Logistic Regression calculates a probability (for example, 0.78 or 78% likelihood of spam), how does it pick a single label?
We apply a Decision Threshold (by default set at 0.5):
- If P(Class = 1) ≥ 0.5 → Predict Class
1(Spam) - If P(Class = 1) < 0.5 → Predict Class
0(Not Spam)
Coach’s Tip: The 0.5 threshold isn’t fixed in stone! In high-risk fields like cancer detection or financial fraud prevention, you might lower the threshold to 0.2. This ensures the model catches more potential positive cases (high sensitivity), even if it creates a few false alarms.
4. Key Comparison: Linear vs. Logistic Regression
| Attribute | Linear Regression | Logistic Regression |
|---|---|---|
| Primary Target | Continuous values (e.g., price, temperature) | Categorical labels (e.g., yes/no, spam/ham) |
| Output Range | Negative infinity to positive infinity $(-\infty, +\infty)$ | Bounded probability $[0.0, 1.0]$ |
| Core Transformation | Straight line / hyperplane fit | S-shaped curve via Sigmoid function |
| Evaluation Metric | Mean Squared Error (MSE), R-squared | Accuracy, Log-Loss (Binary Cross-Entropy), Precision/Recall |
5. Technical Example: Python Implementation
Let’s look at a clean Python script demonstrating how to train a Logistic Regression model using scikit-learn to predict whether a student passes an exam based on study hours:
# Import required libraries
import numpy as np
from sklearn.linear_model import LogisticRegression
# Feature: [Hours Studied]
X = np.array([[1.0], [2.0], [3.0], [4.0], [5.0], [6.0], [7.0], [8.0]])
# Label: [0 = Failed, 1 = Passed]
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
# Initialize and fit the Logistic Regression model
model = LogisticRegression()
model.fit(X, y)
# Predict outcome for a student who studied for 4.5 hours
new_student_hours = np.array([[4.5]])
predicted_class = model.predict(new_student_hours)
predicted_probability = model.predict_proba(new_student_hours)
# Output predictions
print(f"Predicted Class: {predicted_class[0]}")
print(f"Pass Probability: {predicted_probability[0][1]:.2f}")
# Example output:
# Predicted Class: 1
# Pass Probability: 0.62
6. Common Misconceptions
“Why is it called Regression if it does Classification?”
Clarification: Under the hood, the model estimates continuous log-odds (a regression task over log-transformed odds ratios). However, because we wrap that output in a Sigmoid curve and threshold it into discrete groups (0 or 1), it functions as a classification algorithm.
7. Coach’s Note
Note: Never underestimate simple models. Even in sophisticated industry pipelines running modern neural networks, Logistic Regression is frequently deployed as the fast, reliable baseline model. Always build a simple baseline first before reaching for complex deep learning architectures!
8. Quick Recall Checkpoint
Question 1: What is the main role of the Sigmoid function in Logistic Regression?
It maps any unbounded input score into a constrained probability range between 0.0 and 1.0.
Question 2: If a model outputs a probability of 0.75 with a default threshold of 0.5, what label is predicted?
It is predicted as Class 1 (Positive), because 0.75 is greater than or equal to 0.5.
Question 3: Why doesn’t standard Linear Regression work well for binary choices?
Linear Regression can output values far outside the 0 to 1 range and is easily distorted by extreme outliers.
9. Key Takeaways
- Logistic Regression is the foundational machine learning model for binary classification.
- The Sigmoid function transforms linear combination outputs into valid, bounded probability values.
- A Decision Threshold (commonly 0.5) converts predicted probabilities into categorical labels.
- It serves as an essential, computationally lightweight baseline for classification across AI engineering.
What Comes Next?
Today we explored how Logistic Regression draws a smooth linear boundary to separate classes. But what happens when decisions depend on non-linear rules like “If customer age > 30 AND annual income > $50,000”?
Tomorrow on Day 33, we take the next step by exploring Decision Trees Explained — an intuitive algorithm that mirrors step-by-step human decision logic!
Explore the full course syllabus at the 365 Days of AI Course Hub. Prefer studying in Hinglish? Visit the Hinglish Course Edition.