Understanding Large Language Models (LLMs)

# CHAPTER 4

Understanding Large Language Models (LLMs) #

1. Introduction #

2. Learning Objectives #

• Define what a Large Language Model is.

• Understand how AI breaks words into "Tokens".

• Explain the concept and limitations of the "Context Window".

• Comprehend what "Parameters" are and why model size matters.

3. Beginner-Friendly Explanation #

4. What makes it "Large"? #

1. 1. Training Data: They are trained on a substantial portion of the entire public internet (terabytes of text).

1. 2. Parameters: Think of a parameter as a single mathematical connection (a synapse) in the AI's artificial brain. Early models had a few million parameters. Modern models have over 1 Trillion parameters. More parameters generally mean the model is "smarter" and can understand more complex reasoning.

5. Tokens: The Currency of AI #

• A token can be an entire word (e.g., apple).

• A token can be a syllable or chunk of a word (e.g., Ham + bur + ger).

6. The Context Window (Memory Limit) #

• If a model has a Context Window of 8,000 tokens (approx. 6,000 words), and you paste a 10,000-word essay into the prompt, the AI will completely forget the first 4,000 words by the time it reaches the end!

• Modern breakthroughs have pushed Context Windows to massive sizes. Google's Gemini 1.5 Pro boasts a context window of up to 2 Million tokens, allowing you to upload entire textbooks and hour-long videos in a single prompt.

7. JSON Example: How APIs Count Tokens #

{
  "id": "chatcmpl-12345",
  "object": "chat.completion",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Paris is the capital of France."
      }
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 7,
    "total_tokens": 21
  }
}

8. Python Example: Token Estimation #

import tiktoken

# Load the tokenizer used by GPT-4
tokenizer = tiktoken.encoding_for_model("gpt-4")

my_prompt = "Generative AI is absolutely fascinating!"

# Chop the text into numerical tokens
tokens = tokenizer.encode(my_prompt)

print(f"The text was chopped into {len(tokens)} tokens.")
print(f"Token IDs: {tokens}")
# Output: The text was chopped into 7 tokens.
# Token IDs: [40405, 597, 15836, 374, 9821, 34241, 0]

9. Mini Project #

10. Best Practices #

• Mind the Context: When building chatbots, developers must write code that actively truncates or summarizes the conversation history as the chat gets longer, preventing the user from accidentally exceeding the Context Window limit and crashing the app.

11. Common Mistakes #

• Assuming LLMs read letter-by-letter: Because LLMs read "Tokens" (chunks of words), they are notoriously terrible at spelling tasks or counting specific letters. If you ask an LLM, "How many r's are in the word strawberry?", it will often guess incorrectly because it doesn't see "s-t-r-a-w-b-e-r-r-y", it sees the mathematical token ID [49832].

12. Exercises #

1. 1. Explain the difference between an LLM's "Training Data" and its "Context Window".

13. MCQs with Answers #

14. Interview Questions #

• Q: What is a Context Window, and what strategies would you use in software development to ensure a user's chatbot session doesn't exceed it?

• Q: Explain why LLMs sometimes struggle with character-level manipulation (like counting specific letters in a word) due to tokenization.

15. FAQs #

16. Summary #

17. Next Chapter Recommendation #