Classification Algorithms

# CHAPTER 24

Classification Algorithms #

1. Chapter Introduction #

2. Logistic Regression (Binary Classification) #

Despite the word "Regression" in its name, Logistic Regression is used for *Classification*. It predicts the probability (from 0 to 1) that an item belongs to a specific class. If the probability is > 0.5, it predicts "Yes" (1). Otherwise, "No" (0).

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# 1. Initialize
model = LogisticRegression()

# 2. Train (Assuming X_train, y_train are already preprocessed)
# y_train contains 1s (Spam) and 0s (Not Spam)
model.fit(X_train, y_train)

# 3. Predict
predictions = model.predict(X_test)

# 4. Evaluate (What percentage did we get right?)
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy * 100:.1f}%")

3. K-Nearest Neighbors (KNN) #

KNN is intuitive. It plots the new data point, looks at the 'K' closest historical data points, and takes a vote. If K=5, and 4 of the 5 closest points are "Cats", the model classifies the new point as a "Cat".

*Crucial:* KNN relies on distance. You MUST scale your data (StandardScaler) before using KNN, or features with large numbers will dominate the distance calculation.

from sklearn.neighbors import KNeighborsClassifier

# Initialize with K=5
knn_model = KNeighborsClassifier(n_neighbors=5)

# Train and Predict
knn_model.fit(X_train_scaled, y_train)
knn_preds = knn_model.predict(X_test_scaled)

4. Decision Trees #

A Decision Tree works like a flowchart. It asks a series of True/False questions about the features to split the data until it reaches a conclusion. (e.g., *Is Income > 50k?* -> *Is Age > 30?* -> *Classify as 'Will Buy'*).

Decision Trees do *not* require feature scaling!

from sklearn.tree import DecisionTreeClassifier

# Initialize
# max_depth stops the tree from growing infinitely and memorizing the data
tree_model = DecisionTreeClassifier(max_depth=3)

tree_model.fit(X_train, y_train)
tree_preds = tree_model.predict(X_test)

5. Multi-Class Classification #

Classification isn't limited to Yes/No. If your y_train column contains 3 categories (0=Cat, 1=Dog, 2=Bird), Scikit-Learn algorithms automatically handle it. You write the exact same .fit() and .predict() code.

6. Mini Project: Spam Email Classifier #

Let's simulate classifying emails based on two features: Word Count and Number of Links.

import pandas as pd
from sklearn.linear_model import LogisticRegression

# Fake Data
# Target: 1 = Spam, 0 = Safe
X_train = pd.DataFrame({
    'Word_Count': [50, 200, 20, 500, 10],
    'Num_Links': [1, 0, 5, 1, 10]
})
y_train = pd.Series([0, 0, 1, 0, 1]) 

# Train
clf = LogisticRegression()
clf.fit(X_train, y_train)

# Predict a new email: 15 words, 8 links
new_email = pd.DataFrame({'Word_Count': [15], 'Num_Links': [8]})
prediction = clf.predict(new_email)

if prediction[0] == 1:
    print("ALERT: Email classified as SPAM")
else:
    print("Email is SAFE")

7. Common Mistakes #

• Not scaling data for KNN: If Feature A ranges from 0-1 and Feature B ranges from 0-1,000,000, KNN will completely ignore Feature A because the distance calculation is overwhelmed by B. Always use StandardScaler.

• Overfitting Decision Trees: If you don't set a maxdepth, a Decision Tree will keep branching until every single training point is perfectly classified. It will score 100% on training data, and fail miserably on test data because it memorized the noise.

12. Next Chapter Recommendation #