Day 31 • ML Algorithms
Welcome back to 365 Days of AI! Yesterday, on Day 30: How a Machine Learning Model Learns, we explored the inner engine of machine learning—learning parameters, weights, loss functions, and optimization via Gradient Descent.
Today, we are putting those foundational learning principles into practice with our very first machine learning algorithm: Linear Regression.
Whether you are building pricing models, forecasting sales, or laying the groundwork to understand deep neural networks, Linear Regression is the starting point. It is simple, elegant, highly interpretable, and computationally fast.
Coach’s Note: Do not underestimate Linear Regression because of its simplicity. Many world-class AI solutions rely on simple, well-tuned linear models as baseline benchmarks. If you understand how a linear model fits data today, understanding neural networks later in this course will be vastly easier.
What You Will Learn Today
- What Linear Regression is and what core problem it solves.
- The difference between regression and classification tasks.
- The visual and mathematical intuition behind finding the “line of best fit” ($y = wx + b$).
- How Mean Squared Error (MSE) measures prediction errors.
- How to implement Linear Regression in Python using
scikit-learn. - Common pitfalls, misconceptions, and limitations to watch out for.
1. What Is Linear Regression?
At its core, Linear Regression is a supervised machine learning algorithm used to predict a continuous numerical output based on one or more input features.
In machine learning, tasks are broadly divided into two main categories:
- Regression: Predicting a continuous number (e.g., house price, temperature, salary, stock value).
- Classification: Predicting a discrete label or category (e.g., spam vs. not spam, cat vs. dog).
Because Linear Regression predicts continuous outputs, it is a regression algorithm. It works by finding a direct linear relationship between your input variable ($x$) and your target variable ($y$).
2. Mental Model: The Ruler on a Scatter Plot
Mental Model: Finding the Ruler Line
Imagine you have a piece of graph paper covered with scattered points representing houses. The horizontal axis ($x$) represents house size in square feet, and the vertical axis ($y$) represents the price in thousands of dollars.
Now, imagine taking a transparent plastic ruler and placing it over the paper. Your goal is to tilt and slide the ruler until it passes as close to all the data points as possible. Once fixed, you can use that straight line to predict the price of any new house size—even one not plotted on the original graph.
Where the Analogy Breaks: In real-world machine learning, we rarely deal with just one input feature on a two-dimensional graph. When we predict housing prices, we might use 20 features (square footage, bedrooms, age, zip code). Instead of drawing a straight 2D line, Linear Regression constructs a high-dimensional surface (a hyperplane) across multi-dimensional space.
3. The Mathematics: Equation of the Line
You may recall the slope-intercept equation of a straight line from high school algebra:
y = m · x + b
In Machine Learning, we write the exact same equation using slightly different notation:
ŷ = w · x + b
Let’s break down what each symbol means in AI terminology:
| Symbol | Term Name | What It Represents |
|---|---|---|
| ŷ (y-hat) | Predicted Target | The output value predicted by our model for a given input. |
| x | Input Feature | The input data point (e.g., house square footage). |
| w | Weight (Slope) | How heavily $x$ influences $y$. A higher weight means $y$ changes drastically when $x$ changes. |
| b | Bias (Intercept) | The baseline value of $y$ when $x$ is 0. It shifts the line up or down. |
Multiple Linear Regression
When you have multiple features (e.g., size $x_1$, bedrooms $x_2$, age $x_3$), the equation expands naturally:
ŷ = (w₁ · x₁) + (w₂ · x₂) + (w₃ · x₃) + ... + b
4. How the Model Learns: Measuring Error with MSE
How does the computer determine the single best position for the line? It measures how wrong the line is using a Loss Function.
The standard loss function for Linear Regression is Mean Squared Error (MSE):
- For each data point, calculate the residual (the vertical distance between the actual value $y$ and predicted value $\hat{y}$).
- Square each residual (this removes negative signs and penalizes larger errors more heavily).
- Take the average (mean) of all squared errors across the entire dataset.
As we covered on Day 30, the model uses Gradient Descent to tweak weights ($w$) and bias ($b$) iteratively until MSE reaches its lowest possible value. That optimal configuration defines our “line of best fit.”
5. Hands-On Code Example: Python & Scikit-Learn
Let’s write a simple Python script to fit a Linear Regression model using scikit-learn, the industry-standard library for traditional machine learning.
# Step 1: Import required libraries
import numpy as np
from sklearn.linear_model import LinearRegression
# Step 2: Prepare sample data (House size in sq ft vs. Price in $1000s)
# Input features X must be a 2D array
X = np.array([[600], [800], [1000], [1400], [1800], [2200]])
# Target labels y (1D array)
y = np.array([150, 200, 245, 340, 430, 510])
# Step 3: Initialize the Linear Regression model
model = LinearRegression()
# Step 4: Train (fit) the model on our data
model.fit(X, y)
# Step 5: Inspect the learned weight (slope) and bias (intercept)
weight = model.coef_[0]
bias = model.intercept_
print(f"Learned Weight (w): {weight:.4f}")
print(f"Learned Bias (b): {bias:.4f}")
# Step 6: Make a prediction for a new house of 1,200 sq ft
new_house_size = np.array([[1200]])
predicted_price = model.predict(new_house_size)
print(f"Predicted Price for 1,200 sq ft: ${predicted_price[0]:.2f}k")
Code Insight: Notice how simple the standard API is: model.fit(X, y) runs the training algorithm under the hood, and model.predict(new_data) applies the learned equation ŷ = w · x + b to generate new predictions.
6. Common Misconceptions & Pitfalls
1. Assuming Everything is Linear
Linear Regression assumes that the underlying relationship between inputs and output is a straight line. If your data follows a wave pattern or exponential curve, standard linear regression will perform poorly without feature transformations.
2. Sensitivity to Outliers
Because MSE squares prediction errors, a single extreme outlier data point can heavily pull the regression line away from the majority of data points.
3. Correlation vs. Causation
A linear regression model might find a strong weight connecting ice cream sales to sunburn rates. This does not mean buying ice cream causes sunburns; both are influenced by summer temperature. Regression finds correlations, not proof of causation.
7. Mini Exercise: Predict the Output
Let’s test your mental model. Suppose a trained linear regression model has learned the following parameters for predicting a student’s final exam score based on hours studied:
- Weight ($w$): 6.5 (points per hour studied)
- Bias ($b$): 40.0 (baseline score with 0 hours studied)
Question: What exam score will the model predict for a student who studied for 8 hours?
Click to view the step-by-step solution
Apply the formula: ŷ = w · x + b
ŷ = (6.5 · 8) + 40.0
ŷ = 52.0 + 40.0 = 92.0
Answer: The model predicts a score of 92 points.
8. Quick Recall Checkpoint
Test your understanding of today’s key concepts before wrapping up.
1. Is Linear Regression used for classification or regression tasks?
Linear Regression is used for regression tasks because it predicts continuous numeric values (like prices or scores), rather than discrete categories.
2. What is the role of the weight (w) in the linear regression equation?
The weight controls the slope or steepness of the line. It tells us how much the target variable ($y$) changes for every 1-unit increase in the input feature ($x$).
3. Why do we square the errors in Mean Squared Error (MSE)?
Squaring cancels out negative signs (so positive and negative errors don’t sum to zero) and penalizes larger errors more heavily than smaller ones.
Key Takeaways
- Linear Regression predicts continuous numbers by finding a line of best fit ($ŷ = wx + b$).
- Weights ($w$) represent feature importance/slope, while Bias ($b$) shifts the baseline line up or down.
- Mean Squared Error (MSE) measures the distance between model predictions and actual data points.
- Gradient Descent optimizes weights and bias to minimize total MSE.
- Linear Regression is fast, highly interpretable, and serves as an essential baseline model in ML engineering.