Logistic Regression for Classification

# CHAPTER 10

Logistic Regression for Classification #

1. Introduction #

2. Learning Objectives #

• Differentiate between Regression and Classification tasks.

• Understand the concept of Binary Classification.

• Explain how the Sigmoid function outputs probabilities.

• Implement LogisticRegression in Scikit-learn.

• Use predict_proba to analyze model confidence.

3. Classification Basics #

• Binary Classification: Predicting exactly two classes (0 or 1, Spam or Not Spam, Yes or No).

• Multi-class Classification: Predicting three or more classes (Apple, Banana, or Orange).

4. Why Not Linear Regression? #

5. The Sigmoid Function #

• If the output is 0.85, the model is 85% confident the email is Spam.

• If the output is 0.10, the model is 10% confident the email is Spam (meaning it is likely Not Spam).

*Threshold:* By default, Scikit-learn sets a threshold of 0.5. Anything above 0.5 is classified as Class 1. Anything below is Class 0.

6. Mini Project: Spam Detection Model #

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 1. Data (X = [Links, Mistakes], y = [0: Not Spam, 1: Spam])
X = np.array([[1, 0], [2, 1], [10, 5], [12, 8], [3, 0], [15, 6]])
y = np.array([0, 0, 1, 1, 0, 1])

# 2. Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

# 3. Train Model
model = LogisticRegression()
model.fit(X_train, y_train)

# 4. Make Predictions (Outputs 0 or 1)
predictions = model.predict(X_test)

# 5. Evaluate
acc = accuracy_score(y_test, predictions)
print(f"Spam Detection Accuracy: {acc * 100}%")

7. Probability Predictions #

# Predict probabilities for the Test Set
probabilities = model.predict_proba(X_test)

# The output is an array showing [Probability of 0, Probability of 1]
for i in range(len(X_test)):
    print(f"Email {i} - Prob Not Spam: {probabilities[i][0]:.2f} | Prob Spam: {probabilities[i][1]:.2f}")

*Business Use Case:* If the model says an email is 51% likely to be spam, you might send it to the Inbox anyway because false positives annoy users. You might manually set a threshold so it only goes to the Spam folder if the probability is > 0.90.

8. Common Mistakes #

• Forgetting to Scale: Logistic Regression uses an optimizer underneath that solves mathematical gradients. Unscaled data (like a column with values in the millions next to a column with 0s and 1s) will cause the algorithm to converge slowly or fail. Always use StandardScaler.

• Class Imbalance: If your dataset has 99,000 normal transactions and 1,000 fraudulent ones, the model can achieve 99% accuracy by simply predicting "Normal" every time. We will address evaluating imbalanced data in Chapter 16.

9. Best Practices #

• Interpretability: Just like Linear Regression, you can check model.coef in Logistic Regression. This tells you which feature had the highest impact on classifying the data into Class 1.

• Q: In a real-world scenario like medical diagnosis, why might you use .predictproba() to check the raw probability instead of relying on .predict()?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #