LSTM and Sequence Models

# CHAPTER 15

LSTM and Sequence Models #

1. Introduction #

2. Learning Objectives #

• Explain how an LSTM solves the Vanishing Gradient problem.

• Understand the function of the Cell State and internal Gates.

• Implement an LSTM layer in Keras.

• Compare Bi-directional LSTMs to standard LSTMs.

• Build a predictive Sequence Model.

3. How an LSTM Works (The Conveyor Belt) #

4. The Three Gates #

1. 1. Forget Gate: Looks at the new word and the old memory, and decides what old information is no longer relevant and should be thrown away (e.g., the sentence subject changed from "Bob" to "Alice").

1. 2. Input Gate: Decides what *new* information from the current word is important enough to add to the conveyor belt.

1. 3. Output Gate: Decides what the actual output prediction should be for this specific time step.

5. Implementing LSTM in Keras #

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense

# Sentiment Analysis with LSTM
model = Sequential([
    Embedding(input_dim=10000, output_dim=32, input_length=100),
    
    # The LSTM layer with 64 memory units
    LSTM(64),
    
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

*This model will drastically outperform a SimpleRNN on long movie reviews because it can remember the context from the very first sentence!*

6. Bidirectional LSTMs #

from tensorflow.keras.layers import Bidirectional

model_bidir = Sequential([
    Embedding(10000, 32, input_length=100),
    # Wrap the LSTM in a Bidirectional layer
    Bidirectional(LSTM(64)),
    Dense(1, activation='sigmoid')
])

7. Mini Project: Sequence Prediction (Text Generation) #

# Assume X_train is sequences of 10 words, and y_train is the 11th word
# Example X: ["The", "cat", "sat", "on", "the", "mat", "and", "fell", "fast", "asleep"]
# Example y: ["snoring"] -> (represented as an integer ID)

vocab_size = 5000

model_gen = Sequential([
    Embedding(input_dim=vocab_size, output_dim=64, input_length=10),
    
    # Layer 1: LSTM returning sequences
    LSTM(128, return_sequences=True),
    
    # Layer 2: LSTM returning final state
    LSTM(128),
    
    # Output Layer: Softmax across the ENTIRE vocabulary to pick the most likely next word!
    Dense(vocab_size, activation='softmax')
])

model_gen.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

8. Common Mistakes #

• Overfitting with complex LSTMs: LSTMs have millions of parameters (due to all the internal gates). They overfit very quickly on small datasets. Always use Dropout layers or the built-in recurrent_dropout parameter (LSTM(64, dropout=0.2)).

• Ignoring GRUs: TensorFlow also provides a GRU (Gated Recurrent Unit) layer. It is a simplified version of an LSTM that trains much faster and often achieves the exact same accuracy. Always try a GRU first!

9. Best Practices #

• Use 1D Convolutions with LSTMs: A massive industry secret for text processing is passing the Embeddings through a Conv1D and MaxPooling1D layer *before* feeding it to the LSTM. The CNN extracts phrase patterns, shortening the sequence and making the LSTM's job much easier and faster!

10. Exercises #

1. 1. What is the purpose of the "Forget Gate" inside an LSTM cell?

1. 2. Write the Keras code to create an Embedding layer followed by a Bidirectional LSTM layer with 32 units.

11. MCQ Quiz with Answers #

12. Interview Questions #

• Q: Explain the difference in architecture between a SimpleRNN and an LSTM.

• Q: Why would you wrap an LSTM in a Bidirectional wrapper, and in what scenario (like real-time forecasting) would this actually be a bad idea?

13. FAQs #

14. Summary #

15. Next Chapter Recommendation #