Web Scraping with Python

# Web Scraping with Python

Welcome to Chapter 26! Web scraping lets you extract data from websites automatically — collecting prices, news headlines, research data, and more.

---

1. Learning Objectives #

• Understand web scraping concepts and ethics.

• Use requests and BeautifulSoup to scrape HTML.

• Navigate and extract elements from HTML.

• Handle pagination and dynamic content.

• Build a news headline scraper.

---

2. Setup #

pip install requests beautifulsoup4

---

3. Basic Web Scraping #

```python id="py26ex1" import requests from bs4 import BeautifulSoup

# Find all quotes quotes = soup.findall("span", class="text") for i, quote in enumerate(quotes[:5], 1): print(f" {i}. {quote.gettext()}")

---

## 4. Navigating HTML

python id="py26ex2" from bs4 import BeautifulSoup

# Find by class items = soup.findall("li", class_="item") for item in items: print(f" - {item.text}")

# Find link link = soup.find("a") print(f"URL: {link['href']}") # https://python.org print(f"Text: {link.text}") # Python

# CSS selectors items = soup.select("ul.items li.item") print([item.text for item in items])

---

## 5. Extracting Data

python id="py26ex3" import requests from bs4 import BeautifulSoup

# Extract quotes with author and tags quotedivs = soup.findall("div", class="quote")

for div in quotedivs[:3]: text = div.find("span", class="text").gettext() author = div.find("small", class="author").gettext() tags = [tag.gettext() for tag in div.findall("a", class="tag")] print(f" \"{text[:50]}...\"") print(f" — {author}") print(f" Tags: {', '.join(tags)}\n")

---

## 6. Scraping Multiple Pages

python id="py26ex4" import requests from bs4 import BeautifulSoup

allquotes = [] baseurl = "https://quotes.toscrape.com/page/{}/"

for page in range(1, 4): # First 3 pages response = requests.get(baseurl.format(page)) soup = BeautifulSoup(response.text, "html.parser") quotes = soup.findall("span", class="text") for q in quotes: allquotes.append(q.gettext())

print(f"Total quotes collected: {len(allquotes)}") for i, q in enumerate(allquotes[:5], 1): print(f" {i}. {q[:60]}...")

---

## 7. Saving Scraped Data

python id="py26ex5" import csv import json

with open("scrapeddata.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["title", "author", "date"]) writer.writeheader() writer.writerows(data)

# Save to JSON with open("scraped_data.json", "w") as f: json.dump(data, f, indent=2)

print("✅ Data saved to CSV and JSON!")

---

## 8. Ethical Scraping Guidelines

1. **Check `robots.txt`** — `website.com/robots.txt` tells what's allowed.
2. **Respect rate limits** — Add delays between requests.
3. **Don't overload servers** — Use `time.sleep()`.
4. **Check terms of service** — Some sites prohibit scraping.
5. **Use APIs when available** — They're the intended data access method.

python id="py26_ex6" import time

# Polite scraping with delays for page in range(1, 5): # response = requests.get(url) time.sleep(2) # Wait 2 seconds between requests

---

## 9. Mini Project: News Headline Scraper

python id="py26project" import requests from bs4 import BeautifulSoup

def scrapequotesasnews(): """Scrape quotes site as a demo (reliable for learning)""" url = "https://quotes.toscrape.com/" try: response = requests.get(url, timeout=10) response.raiseforstatus() soup = BeautifulSoup(response.text, "html.parser") print("=" * 50) print(" 📰 HEADLINE SCRAPER (Demo)") print("=" * 50) quotes = soup.findall("div", class="quote") for i, quotediv in enumerate(quotes, 1): text = quotediv.find("span", class="text").gettext() author = quotediv.find("small", class="author").gettext() tags = [t.gettext() for t in quotediv.findall("a", class="tag")] print(f"\n [{i}] {text[:70]}...") print(f" Author: {author}") print(f" Tags: {', '.join(tags[:3])}") print(f"\n {'='*50}") print(f" Total items scraped: {len(quotes)}") except requests.exceptions.RequestException as e: print(f" ❌ Error: {e}")

scrapequotesasnews() ``

Q2: soup.findall() returns: A) First match B) List of matches C) Boolean D) String Answer: B

Q5: To get attribute from tag <a href="url">: A) tag.href B) tag["href"] C) tag.getattr() D) tag.href() Answer: B

Q7: gettext() returns: A) HTML B) Plain text without tags C) Attributes D) Tag name Answer: B

• Use find(), findall(), and select()` to navigate HTML.

• Save scraped data to CSV or JSON.

• Always practice ethical scraping — check robots.txt, add delays.

• Use Selenium for JavaScript-rendered content.

---

13. Next Chapter Recommendation #

In Chapter 27: Python Automation Projects, you'll automate file management, emails, and more! 🚀