Model Evaluation Metrics for Regression

# CHAPTER 17

Model Evaluation Metrics for Regression #

1. Introduction #

2. Learning Objectives #

• Define a "Residual" (Error).

• Calculate and interpret Mean Absolute Error (MAE).

• Calculate and interpret Mean Squared Error (MSE).

• Understand why Root Mean Squared Error (RMSE) is the industry standard.

• Interpret the $R^2$ (R-Squared) Score.

• Analyze Residual plots to detect model flaws.

3. What is a Residual? #

• True House Price: $300,000

• Model Prediction: $280,000

• Residual (Error): $20,000

To grade a model, we must aggregate the residuals of *every single prediction* it made on the Test Set into a single number.

4. Mean Absolute Error (MAE) #

Interpretation: *"On average, our model's predictions are off by $20,000."*

from sklearn.metrics import mean_absolute_error

# Assuming y_test are the real answers, and predictions are the model's guesses
# mae = mean_absolute_error(y_test, predictions)
# print(f"MAE: ${mae:.2f}")

5. Mean Squared Error (MSE) #

• *Why square them?* Squaring mathematically punishes massive mistakes heavily. If an error is 2, the square is 4. If an error is 10, the square is 100!

• *The Problem:* The resulting number is no longer in dollars; it is in "squared dollars," which makes no logical sense to a human.

from sklearn.metrics import mean_squared_error

# mse = mean_squared_error(y_test, predictions)

6. Root Mean Squared Error (RMSE) #

import numpy as np

# rmse = np.sqrt(mean_squared_error(y_test, predictions))
# print(f"RMSE: ${rmse:.2f}")

7. R-Squared ($R^2$) Score #

• $R^2 = 1.0$ (100%): A perfect model. It correctly predicted every single dot.

• $R^2 = 0.0$ (0%): A terrible model. It is no better than just guessing the average price every time.

• $R^2 < 0.0$ (Negative): A catastrophic model. It is mathematically worse than just guessing the average!

from sklearn.metrics import r2_score

# r2 = r2_score(y_test, predictions)
# print(f"R-Squared Score: {r2 * 100:.1f}%")

8. Mini Project: Compare Two Models #

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, r2_score

# Mock Test Data
y_true = np.array([100, 150, 200, 250, 300])

# Model 1 Predictions (Linear)
preds_linear = np.array([110, 140, 210, 240, 320])

# Model 2 Predictions (Tree)
preds_tree = np.array([105, 145, 195, 255, 305])

# Evaluate Linear Model
rmse_linear = np.sqrt(mean_squared_error(y_true, preds_linear))
r2_linear = r2_score(y_true, preds_linear)

# Evaluate Tree Model
rmse_tree = np.sqrt(mean_squared_error(y_true, preds_tree))
r2_tree = r2_score(y_true, preds_tree)

print(f"Linear -> RMSE: {rmse_linear:.2f} | R2: {r2_linear:.4f}")
print(f"Tree   -> RMSE: {rmse_tree:.2f} | R2: {r2_tree:.4f}")

# The Tree model will have a lower RMSE and higher R2. It wins!

9. Common Mistakes #

• Evaluating on the Training Data: If you calculate RMSE by asking the model to predict the data it was trained on, it will achieve a near-perfect score. This is cheating. You MUST calculate metrics on a separate ytest holdout set that the model has never seen.

• Residual Analysis: Calculate residuals = ytrue - predictions and plot them on a scatter plot. If the dots are a random cloud around the 0-line, your model is healthy. If the dots form a U-shape, your model missed a massive non-linear pattern!

11. Exercises #

1. 1. If your model's MAE is $5,000 and its RMSE is $45,000, what does this tell you about the types of mistakes the model is making?

1. 2. What does an R-Squared score of -0.5 mean?

12. MCQ Quiz with Answers #

13. Interview Questions #

• Q: Explain the mathematical and practical difference between Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). In what scenario would one be vastly different from the other?

• Q: Describe how you would use a Residual Plot to detect Heteroscedasticity in a trained model.

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #