Natural Language Processing Basics with PyTorch

# CHAPTER 13

Natural Language Processing Basics with PyTorch #

1. Introduction #

2. Learning Objectives #

• Explain the standard NLP preprocessing workflow.

• Understand Tokenization and Vocabulary creation.

• Convert sentences into numerical sequences.

• Pad sequences to handle variable sentence lengths.

• Understand and implement nn.Embedding in PyTorch.

3. The NLP Workflow #

1. 1. Cleaning: Removing punctuation, HTML tags, and making everything lowercase.

1. 2. Tokenization: Splitting a sentence into individual words (Tokens).

1. 3. Vocabulary Building: Assigning a unique integer to every unique word in your entire dataset (e.g., "I" = 1, "love" = 2).

1. 4. Sequencing & Padding: Replacing words with their integer IDs. Since neural networks require fixed-size inputs, if one sentence is 5 words and another is 10, we add 0s to the short sentence to make them equal length.

4. Step-by-Step: Tokenization and Sequencing #

import torch

# 1. Mock Dataset
sentences = [
    "I love deep learning",
    "I love PyTorch",
    "Deep learning is amazing"
]

# 2. Tokenization & Vocabulary Building
vocab = {"<PAD>": 0, "<OOV>": 1} # PADDING and Out-Of-Vocabulary tokens
word_idx = 2

for sentence in sentences:
    for word in sentence.lower().split():
        if word not in vocab:
            vocab[word] = word_idx
            word_idx += 1

print("Vocabulary:", vocab)
# Output: {'<PAD>': 0, '<OOV>': 1, 'i': 2, 'love': 3, 'deep': 4, 'learning': 5, ...}

# 3. Sequencing and Padding
def text_to_tensor(text, vocab, max_len=5):
    sequence = [vocab.get(w, vocab["<OOV>"]) for w in text.lower().split()]
    # Padding: Add 0s to the end if it's too short
    while len(sequence) < max_len:
        sequence.append(vocab["<PAD>"])
    # Truncate if it's too long
    return torch.tensor(sequence[:max_len])

tensor_seq = text_to_tensor("I love PyTorch", vocab)
print("Sequence Tensor:", tensor_seq)
# Output: tensor([2, 3, 6, 0, 0]) -> Padded perfectly!

5. The Problem with Integers #

6. Word Embeddings (nn.Embedding) #

• "Apple" (the fruit) might be plotted at [0.1, 0.9].

• "Banana" might be plotted at [0.1, 0.95].

7. Adding an Embedding Layer in PyTorch #

import torch.nn as nn

class NLPModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim):
        super().__init__()
        
        # 1. The Embedding Layer
        # num_embeddings = Total words in dictionary
        # embedding_dim = How many dimensions (axes) the semantic graph has
        self.embedding = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_dim)
        
        # 2. Flatten the embeddings so Linear layers can read them
        self.flatten = nn.Flatten()
        
        # 3. Dense classification layers (Assuming sequences are exactly 5 words long)
        self.fc = nn.Linear(in_features=embedding_dim * 5, out_features=16)
        self.relu = nn.ReLU()
        self.output = nn.Linear(16, 1) # Binary sentiment (Positive or Negative)

    def forward(self, x):
        # x starts as integers: [2, 3, 6, 0, 0]
        x = self.embedding(x) 
        # Now x is a massive matrix of decimals containing semantic meaning!
        x = self.flatten(x)
        x = self.relu(self.fc(x))
        x = self.output(x)
        return x

model = NLPModel(vocab_size=len(vocab), embedding_dim=8)
print(model)

8. Common Mistakes #

• Forgetting the <OOV> token: When your model is deployed, users will type words that weren't in your training data. If you don't use an Out-Of-Vocabulary (<OOV>) token, your code will crash when it tries to look up a word that doesn't exist in the dictionary.

• Applying Embeddings to Floats: The input to an nn.Embedding layer must be integer IDs (dtype=torch.long). If you pass floats, PyTorch will throw an error because it uses these IDs as exact row indexes in a lookup table.

9. Best Practices #

• Pre-trained Embeddings: Just like Transfer Learning for images, you don't have to train Embeddings from scratch. You can download pre-trained Embeddings (like GloVe or Word2Vec) that already know the relationships between billions of English words, and load them directly into your nn.Embedding layer!

10. Exercises #

1. 1. What does padding a sequence mean, and why is it mandatory for Neural Networks?

1. 2. Explain why representing the word "Dog" as the integer 5 and "Cat" as the integer 10 causes mathematical problems in a standard Linear network.

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain the step-by-step preprocessing pipeline required to feed a raw text string into a PyTorch neural network.

• Q: What is an Out-Of-Vocabulary (OOV) token, and why is it crucial for production NLP models?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #