Feature Engineering and Data Preprocessing

# CHAPTER 13

Feature Engineering and Data Preprocessing #

1. Introduction #

2. Learning Objectives #

• Explain the concept of Feature Engineering.

• Encode categorical text data using One-Hot Encoding.

• Understand the mathematical need for Feature Scaling.

• Implement StandardScaler to normalize data.

• Prevent Data Leakage during preprocessing.

3. What is Feature Engineering? #

One-Hot Encoding creates a new binary (1 or 0) column for every unique category.

import pandas as pd

# Raw data with text categories
df = pd.DataFrame({
    "Age": [25, 30, 22],
    "Sub_Type": ["Basic", "Pro", "Enterprise"]
})

# Apply One-Hot Encoding using Pandas
# drop_first=True prevents the Dummy Variable Trap!
df_encoded = pd.get_dummies(df, columns=["Sub_Type"], drop_first=True)
print(df_encoded)

# Output:
#    Age  Sub_Type_Enterprise  Sub_Type_Pro
# 0   25                    0             0   <- (This means they are Basic!)
# 1   30                    0             1
# 2   22                    1             0

5. Why Do We Need Feature Scaling? #

• WebsiteVisits: Ranges from 1 to 10.

• AnnualIncome: Ranges from $30,000 to $150,000.

For algorithms that calculate physical distance (like KNN or SVM) or use gradient descent (like Logistic Regression), the massive numbers in the Income column will mathematically drown out the Visits column. The algorithm will falsely assume Income is 10,000 times more important. Feature Scaling squashes all columns down to the exact same numerical scale so the algorithm treats them fairly.

6. Standardization (StandardScaler) #

from sklearn.preprocessing import StandardScaler
import numpy as np

# Raw data: massive differences in scale
X_train = np.array([
    [5, 40000],  # 5 visits, $40k income
    [2, 35000], 
    [10, 120000]
])

scaler = StandardScaler()

# fit_transform calculates the mean/variance AND scales the data
X_scaled = scaler.fit_transform(X_train)

print(X_scaled)
# Both columns are now centered around 0 with similar spreads!
# [[-0.2  -0.54]
#  [-1.13 -0.68]
#  [ 1.33  1.22]]

7. The Golden Rule: Prevent Data Leakage #

CRITICAL RULE: You must fit your StandardScaler ONLY on the Training Data. You then apply that exact same scaler to the Test Data. Never fit_transform the entire dataset before splitting it!

# CORRECT PREPROCESSING WORKFLOW
from sklearn.model_selection import train_test_split

# 1. Split first!
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# 2. Fit the scaler ONLY on the Training data to learn its Mean/Max
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

# 3. ONLY TRANSFORM the Test data using the rules learned from Training data
X_test_scaled = scaler.transform(X_test)

8. Common Mistakes #

• Scaling Decision Trees: As covered in Chapter 8, Decision Trees and Random Forests do NOT require Feature Scaling. Applying a StandardScaler to data before feeding it to a Random Forest is a waste of CPU time, though it won't hurt the accuracy.

• Label Encoding non-ordinal data: If you convert "Red", "Green", "Blue" into 1, 2, 3, the algorithm will mathematically assume Blue (3) is "greater" than Red (1). This is false. Use One-Hot Encoding for categories without a natural order.

9. Best Practices #

• Drop useless columns: If a feature has no logical connection to the target (like a CustomerID or a random Timestamp), drop it. Garbage data causes overfitting.

1. 2. Explain the difference between .fittransform() and .transform() in Scikit-Learn, and when you should use each.

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain "Data Leakage" in the context of applying a StandardScaler. How do you architect your code to prevent it?

• Q: Describe a scenario where creating a new feature mathematically (Feature Engineering) would significantly improve a model's classification performance compared to using the raw data.

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #