Performance Optimization in Jupyter

# CHAPTER 18

Performance Optimization in Jupyter #

1. Chapter Introduction #

2. The Kernel Death (Out of Memory Error) #

If you see a message saying "The kernel appears to have died. It will restart automatically.", 99% of the time, you ran out of RAM (Memory).

How to check your memory usage: Run this magic command to see what variables are hogging your RAM:

%whos

This prints a table showing variable names, their type, and the data/size they contain.

3. Deleting Variables to Free RAM #

If you load a massive DataFrame, process it, and create a *new* cleaned DataFrame, the old messy one is still sitting in memory taking up gigabytes of space. You must explicitly delete it.

Cell 1:

import pandas as pd

# Load massive file
raw_data = pd.read_csv('massive_file.csv')

# Process it
clean_data = raw_data.dropna()

# Delete the raw_data variable from the Kernel memory
del raw_data

# Force Python's Garbage Collector to free the RAM immediately
import gc
gc.collect()

4. Pandas Memory Optimization #

Pandas uses 64-bit precision for numbers by default, which is overkill for most data.

Cell 2:

# Create sample data
df = pd.DataFrame({'age': [25, 30, 35, 40], 'status': ['Active', 'Active', 'Inactive', 'Active']})

print("Original Memory:")
df.info(memory_usage='deep')

# 1. Downcast Integers (Ages don't need 64 bits, 8 bits can store up to 127)
df['age'] = pd.to_numeric(df['age'], downcast='integer')

# 2. Convert text to Categories (Stores 'Active' once, maps it to integers)
df['status'] = df['status'].astype('category')

print("\nOptimized Memory:")
df.info(memory_usage='deep')
# Memory drops significantly!

5. Handling Massive Datasets (Chunking) #

If a CSV is 20GB and your RAM is 8GB, you cannot use pd.read_csv(). You must read the file in small "chunks".

Cell 3:

# Process 100,000 rows at a time
chunk_size = 100000

# This returns an iterator, not a DataFrame
chunk_iterator = pd.read_csv('massive_file.csv', chunksize=chunk_size)

total_revenue = 0

for chunk in chunk_iterator:
    # Do math on this small chunk
    chunk_revenue = chunk['price'].sum()
    total_revenue += chunk_revenue
    
    # The chunk is discarded from memory when the loop repeats!

print(f"Total Revenue: {total_revenue}")

6. Avoiding Print on Massive Arrays #

If you accidentally type a variable name representing a list of 10 million items at the end of a cell, Jupyter will try to render all of it to the screen. This will freeze your web browser, even if your computer has plenty of RAM.

*Always* use .head() for DataFrames, or slice lists like massive_list[:10] to only preview the first 10 items.

7. Profiling Execution Time #

If your notebook is running slowly, find the bottleneck using the %time and %timeit magic commands.

Cell 4:

import numpy as np

# %time runs the code ONCE and tells you how long it took
%time sum(range(1000000))

# %timeit runs the code multiple times to get an accurate average
%timeit np.sum(np.arange(1000000))

8. Common Mistakes #

• Keeping hidden copies of data: Doing df2 = df.copy() doubles your memory usage instantly. Unless you strictly need a separate copy to modify, avoid duplicating DataFrames.

• Ignoring the del keyword: In standard scripts, when a script finishes, memory is cleared. In Jupyter, memory is held forever until you restart the Kernel. You must actively use del variablename during long sessions.

10. Interview Questions #

• Q: Your Jupyter kernel crashes every time you try to load a specific CSV file. Describe three strategies you could use to process this file successfully on your laptop.

• Q: Explain how the Python Garbage Collector interacts with the Jupyter Kernel.

11. Summary #

12. Next Chapter Recommendation #