Logistic Regression for Classification

# CHAPTER 6

Logistic Regression for Classification #

1. Introduction #

2. Learning Objectives #

• Explain why standard Linear Regression fails at classification.

• Understand the mathematics of the Sigmoid Function.

• Train a LogisticRegression model using scikit-learn.

• Output probability scores using .predict_proba().

• Understand Decision Thresholds.

3. Why Linear Regression Fails #

4. The Math: The Sigmoid Function #

*The Result:* The algorithm no longer outputs nonsense like 3.5. It outputs a Probability. If the model outputs 0.85, it means it is 85% confident the item belongs to Class 1.

5. Decision Thresholds #

• If Probability $\ge 0.50 \rightarrow$ Predict Class 1.

• If Probability $< 0.50 \rightarrow$ Predict Class 0.

6. Mini Project: Email Spam Detection #

import numpy as np
from sklearn.linear_model import LogisticRegression

# 1. Provide the Data
# Features (X): [Number of Links, Email Length in words]
X_train = np.array([
    [1, 50],   # Safe
    [0, 200],  # Safe
    [5, 30],   # Spam (Lots of links, short text)
    [8, 20]    # Spam
])

# Labels (y): 0 = Safe, 1 = Spam
y_train = np.array([0, 0, 1, 1])

# 2. Initialize the Model
model = LogisticRegression()

# 3. Train the Model
model.fit(X_train, y_train)

# 4. Make a Prediction!
# New email arrives: 6 links, 25 words long
X_test = np.array([[6, 25]])

# Get the hard class prediction (0 or 1)
prediction = model.predict(X_test)
print(f"Predicted Class: {prediction[0]}") # Output: 1 (Spam!)

7. Extracting Probabilities (predictproba) #

# Extract the raw probabilities
probabilities = model.predict_proba(X_test)

# predict_proba returns an array with probabilities for BOTH classes: [Prob of 0, Prob of 1]
print(f"Probability of Safe (Class 0): {probabilities[0][0]*100:.1f}%")
print(f"Probability of Spam (Class 1): {probabilities[0][1]*100:.1f}%")

# Example Output:
# Probability of Safe (Class 0): 12.5%
# Probability of Spam (Class 1): 87.5%

*Because 87.5% is greater than the 50% threshold, the model returned Class 1!*

8. Common Mistakes #

• Assuming Logistic Regression draws curved boundaries: Despite using the curved Sigmoid function for probabilities, the actual physical Decision Boundary that Logistic Regression draws through the data is a perfectly straight line. If your data cannot be separated by a straight line, Logistic Regression will underfit.

• Ignoring the Threshold: In a medical scenario (detecting cancer), you don't want to wait until the model is 50% sure. You might want to flag the patient if the model is even 15% sure! You can manually extract the probabilities using predictproba and write your own custom if prob > 0.15 logic to override the default 50% threshold.

1. 1. If .predictproba() outputs [0.30, 0.70], what hard class will .predict() output, assuming the default threshold?

1. 2. Why is a standard linear line ($y = mx+b$) mathematically inappropriate for predicting binary classes like 0 or 1?

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain how a Logistic Regression model utilizes a Threshold to convert its raw mathematical output into a final class prediction.

• Q: In what specific business scenario would you manually lower the decision threshold of a Logistic Regression model from 0.50 to 0.10?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #