Hyperparameter Tuning and Optimization

# CHAPTER 18

Hyperparameter Tuning and Optimization #

1. Introduction #

2. Learning Objectives #

• Explain what an Optimizer does mathematically.

• Compare SGD (Stochastic Gradient Descent) vs. Adam.

• Understand the critical importance of the Learning Rate.

• Implement Learning Rate Schedulers in PyTorch.

• Apply Regularization (Weight Decay) to prevent overfitting.

3. What is an Optimizer? #

4. The Learning Rate (The Most Important Hyperparameter) #

• LR Too Large (e.g., 0.1): You take massive leaps. You might jump completely over the valley and end up higher on the other side. The model fails to converge and the loss fluctuates wildly.

• LR Too Small (e.g., 0.000001): You take microscopic baby steps. It will take you 10 years to reach the bottom. Training takes forever, or the model gets stuck in a small ditch (local minima).

• Just Right (Usually 0.001 or 0.0001): You walk smoothly down to the lowest point.

5. Optimizers: SGD vs. Adam #

• SGD (Stochastic Gradient Descent): The classic algorithm. It is simple, reliable, but can be slow and requires you to manually pick the perfect Learning Rate.

• Adam (Adaptive Moment Estimation): The modern industry standard. Adam doesn't use one static Learning Rate. It *adapts* the learning rate for every single weight dynamically during training. It starts fast and slows down as it gets closer to the valley. Always use Adam as your baseline.

import torch.optim as optim

# Standard Adam setup
optimizer_adam = optim.Adam(model.parameters(), lr=0.001)

# Standard SGD setup (Momentum helps push it out of small ditches)
optimizer_sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

6. Learning Rate Scheduling #

# Create Optimizer
optimizer = optim.Adam(model.parameters(), lr=0.01)

# Create Scheduler: Reduce LR by 10x every 5 epochs
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

for epoch in range(20):
    # ... Standard training steps (forward, backward, step) ...
    
    # CRITICAL: Step the scheduler at the END of the epoch
    scheduler.step()
    
    # Print the current learning rate
    current_lr = scheduler.get_last_lr()[0]
    print(f"Epoch {epoch} | Current LR: {current_lr}")

7. Regularization (Weight Decay) #

# weight_decay adds a mathematical penalty for large weights, forcing the network to stay simple
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)

8. Automated Hyperparameter Search #

9. Common Mistakes #

• Stepping the Scheduler at the wrong time: You must call scheduler.step() at the *end* of the Epoch loop, NOT inside the batch loop. Stepping it inside the batch loop will cause your Learning Rate to hit zero in a matter of seconds.

• Changing too many things at once: If you change the architecture, the optimizer, and the batch size all at the same time, and accuracy drops, you don't know which change caused it. Treat tuning like a science experiment: change one variable at a time.

10. Best Practices #

• Use Logarithmic Scales for LR: When testing learning rates, don't test 0.01 and 0.02. Test by orders of magnitude (Powers of 10): 0.1, 0.01, 0.001, 0.0001.

11. Exercises #

1. 1. What happens to the training process if the Learning Rate is set astronomically high (e.g., 100.0)?

1. 2. Explain the difference between optimizer.step() and scheduler.step().

12. MCQ Quiz with Answers #

13. Interview Questions #

• Q: Explain the metaphor of walking down a mountain in relation to the Optimizer and the Learning Rate.

• Q: What is the purpose of Weight Decay (L2 Regularization), and how is it implemented in PyTorch?

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #