Model Evaluation Metrics for Classification

# CHAPTER 16

Model Evaluation Metrics for Classification #

1. Introduction #

2. Learning Objectives #

• Deconstruct the 4 quadrants of a Confusion Matrix.

• Calculate and interpret Precision.

• Calculate and interpret Recall (Sensitivity).

• Understand the F1-Score harmonic mean.

• Analyze an ROC Curve and interpret the AUC score.

3. The Confusion Matrix #

Assume Class 1 = "Has Cancer", Class 0 = "Healthy".

• True Positives (TP): Model predicted Cancer (1), and the patient actually had Cancer (1). *SUCCESS!*

• True Negatives (TN): Model predicted Healthy (0), and the patient was actually Healthy (0). *SUCCESS!*

• False Positives (FP) [Type I Error]: Model predicted Cancer (1), but the patient was Healthy (0). *FALSE ALARM.*

• False Negatives (FN) [Type II Error]: Model predicted Healthy (0), but the patient actually had Cancer (1). *CATASTROPHE.*

from sklearn.metrics import confusion_matrix

# y_test = Real answers, predictions = Model guesses
# print(confusion_matrix(y_test, predictions))

4. Precision (Quality of the Alarm) #

• Formula: TP / (TP + FP)

• Interpretation: If a Spam Filter has 99% Precision, it means when it flags an email as Spam, you can trust it. It generates very few False Alarms (False Positives).

5. Recall / Sensitivity (Catching the Bad Guys) #

• Formula: TP / (TP + FN)

• Interpretation: If a Cancer detector has 99% Recall, it means it caught almost every sick patient. It generated very few False Negatives.

*The Tradeoff:* To get 100% Recall, a model can just predict "Cancer" for everyone. It caught all the cancer, but generated a million False Positives (terrible Precision). You must balance them!

6. The F1-Score (The Ultimate Metric) #

7. Mini Project: Generate a Classification Report #

import numpy as np
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.linear_model import LogisticRegression

# Mock Test Data
y_test = np.array([0, 0, 0, 0, 1, 1, 1])
# Model made some mistakes!
predictions = np.array([0, 0, 0, 1, 1, 1, 0])

print("--- CONFUSION MATRIX ---")
print(confusion_matrix(y_test, predictions))
# Output:
# [[3  1]   <- 3 True Negatives, 1 False Positive
#  [1  2]]  <- 1 False Negative, 2 True Positives

print("\n--- CLASSIFICATION REPORT ---")
print(classification_report(y_test, predictions))
# Output will display Precision, Recall, and F1-Score for BOTH classes!

8. The ROC Curve and AUC Score #

• AUC = 1.0: A perfect model.

• AUC = 0.5: A worthless model. It is exactly the same as flipping a coin.

9. Common Mistakes #

• Optimizing the wrong metric for the business: In Cancer detection, a False Negative (missing the cancer) is lethal. You must optimize for Recall. In Spam filtering, a False Positive (sending an important email from the CEO to the Spam folder) is lethal. You must optimize for Precision.

• Using Accuracy on 99/1 Imbalanced Data: As drilled in Chapter 14, standard Accuracy is a complete mathematical lie when classes are skewed.

10. Best Practices #

• Adjust the Threshold: If you need 99% Recall (catch all the fraud), don't accept Scikit-Learn's default 0.5 threshold. Extract the probabilities using .predictproba() and write logic to predict Class 1 if the probability is even > 0.15!