Polynomial Regression

# CHAPTER 11

Polynomial Regression #

1. Introduction #

2. Learning Objectives #

• Identify non-linear relationships in data.

• Understand the math behind Polynomial terms ($x^2$, $x^3$).

• Transform raw features using PolynomialFeatures.

• Fit a Polynomial Regression model in scikit-learn.

• Recognize the severe risks of Overfitting with high-degree polynomials.

3. The Math: Bending the Line #

To make the line curve, we simply add a "squared" version of our input feature to the equation! $y = (m1 \times x) + (m2 \times x^2) + b$

This is a Degree-2 Polynomial (a Parabola/U-shape). If the data wiggles twice (like an S-shape), we add a cubed term: $y = (m1 \times x) + (m2 \times x^2) + (m3 \times x^3) + b$

*The magic:* Even though the line is curving, the algorithm to find the weights ($m1, m2$) is the exact same Linear Regression algorithm! We are just feeding it engineered features.

4. Mini Project: Student Score Prediction Curve #

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# 1. The Curved Data
X = np.array([[1], [2], [3], [4], [5], [6], [7]]) # Hours Studied
y = np.array([45, 50, 60, 80, 85, 88, 90])       # Exam Score

# 2. Transform the Data! (The Magic Step)
# We tell scikit-learn to create the x^2 column
poly_transformer = PolynomialFeatures(degree=2)
X_poly = poly_transformer.fit_transform(X)

# Look at the transformed data: It created [1, x, x^2]
# print(X_poly)

# 3. Fit a STANDARD Linear Regression model on the NEW polynomial data
model = LinearRegression()
model.fit(X_poly, y)

# 4. Visualize the Curve
# Create 100 smooth points between 1 and 7 for a smooth line
X_smooth = np.linspace(1, 7, 100).reshape(-1, 1)
X_smooth_poly = poly_transformer.transform(X_smooth)
y_smooth_pred = model.predict(X_smooth_poly)

plt.scatter(X, y, color='blue', label='Actual Scores')
plt.plot(X_smooth, y_smooth_pred, color='red', label='Polynomial Curve')
plt.title('Hours Studied vs Exam Score (Degree=2)')
plt.legend()
plt.show()

5. The Danger of Degrees (Overfitting) #

• Degree = 1: A straight line. (Underfitting a curve).

• Degree = 2 or 3: A smooth, logical curve. (The Sweet Spot).

• Degree = 15: The model creates a hyper-complex, chaotic squiggle that perfectly touches every single data point, but goes wildly off the chart in between points. This is extreme Overfitting (High Variance). It has memorized the training data and will fail miserably in the real world.

6. Pipeline Implementation #

from sklearn.pipeline import make_pipeline

# Create a single object that does the polynomial transformation AND fits the model
poly_model = make_pipeline(PolynomialFeatures(degree=3), LinearRegression())

# Train it directly on the raw X data!
poly_model.fit(X, y)

7. Common Mistakes #

• Forgetting to transform Test Data: If you use PolynomialFeatures on your Training Data, your model is now trained to expect 3 columns ($1, x, x^2$). If you try to run model.predict() on raw Test Data that only has 1 column ($x$), the code will crash. You MUST transform the test data first, or use a Pipeline!

• Extrapolation is catastrophic: Linear regression goes straight off the chart. Polynomial regression curves off the chart. If you predict data far outside your training range with a Degree-3 polynomial, the prediction will likely skyrocket to negative infinity or millions.

8. Best Practices #

• Keep Degrees Low: In 99% of real-world scenarios, a Degree of 2 or 3 is the maximum you should ever use. If your data requires a Degree of 10 to fit, you should be using a completely different algorithm like a Random Forest.

9. Exercises #

1. 1. If your raw input feature $X$ is [3], what will the output array look like after passing it through PolynomialFeatures(degree=3)? (Hint: don't forget the bias column of 1s).

1. 2. Write a Python snippet using make_pipeline to create a Degree-4 polynomial regression model.

10. MCQ Quiz with Answers #

11. Interview Questions #

• Q: Explain how setting the degree hyperparameter relates to the Bias-Variance tradeoff in Polynomial Regression.

• Q: Why is Extrapolation (predicting outside the training range) particularly dangerous when using Polynomial Regression compared to Simple Linear Regression?

12. FAQs #

13. Summary #

14. Next Chapter Recommendation #