Web Crawling Basics — BeautifulSoup

After completing this topic

You will be able to implement the basic flow of fetching web pages with requests and extracting desired data with BeautifulSoup.

What is Web Crawling?

Web crawling is the process of a program visiting web pages and automatically collecting data. It is also known as scraping.

It's essentially performing the same tasks a browser does in code:

Sending a request to a web server.
Receiving HTML.
Extracting the desired data from the HTML.

Step 1: Fetching HTML (requests)

python

import requests

url = "https://example.com"
response = requests.get(url)

print(response.status_code)  # 200 = OK
print(response.text[:200])   # The first 200 characters of the HTML text

requests.get(url) sends a request to the server, just like a browser, and receives the HTML sent by the server in response.text.

python

# Check the status code
if response.status_code == 200:
    html = response.text
elif response.status_code == 404:
    print("Page not found")
elif response.status_code == 403:
    print("Access denied")

Step 2: Parsing HTML (BeautifulSoup)

python

from bs4 import BeautifulSoup

html = """
<html>
<body>
  <h1 class="title">News Title</h1>
  <div class="content">
    <p>This is the first paragraph.</p>
    <p>This is the second paragraph.</p>
  </div>
  <ul id="tags">
    <li>Python</li>
    <li>Data</li>
    <li>Crawling</li>
  </ul>
</body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

BeautifulSoup parses the HTML string and creates a tree-structured object. You can then use this object to find the desired parts by tag, class, or ID.

Step 3: Extracting Data

Finding by Tag

python

# The first h1 tag
title = soup.find("h1")
print(title.text)         # "News Title"
print(title["class"])     # ["title"]

# All p tags
paragraphs = soup.find_all("p")
for p in paragraphs:
    print(p.text)
# "This is the first paragraph."
# "This is the second paragraph."

Finding by CSS Selector

python

# select — Uses CSS selectors (equivalent to querySelectorAll)
items = soup.select("ul#tags li")
for item in items:
    print(item.text)
# "Python"
# "Data"
# "Crawling"

# Select by class
content = soup.select_one("div.content")
print(content.text.strip())

select/select_one is more intuitive for those familiar with CSS than find/find_all.

Practical Example: Extracting Table Data

python

table_html = """
<table>
  <tr><th>Name</th><th>Score</th></tr>
  <tr><td>Cheol-soo</td><td>85</td></tr>
  <tr><td>Young-hee</td><td>92</td></tr>
  <tr><td>Min-soo</td><td>78</td></tr>
</table>
"""

soup = BeautifulSoup(table_html, "html.parser")
rows = soup.select("tr")

data = []
for row in rows[1:]:  # Exclude header
    cols = row.find_all("td")
    data.append({
        "Name": cols[0].text,
        "Score": int(cols[1].text)
    })

print(data)
# [{'Name': 'Cheol-soo', 'Score': 85}, {'Name': 'Young-hee', 'Score': 92}, {'Name': 'Min-soo', 'Score': 78}]

python

# Convert directly to pandas DataFrame
import pandas as pd
df = pd.DataFrame(data)
print(df)

Points to Note

Request Interval

python

import time

urls = ["https://example.com/page/1", "https://example.com/page/2"]
for url in urls:
    response = requests.get(url)
    time.sleep(1)  # Wait for 1 second - to avoid overloading the server

Sending requests too quickly can overload the server or block your IP address.

Setting User-Agent

python

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/91.0"
}
response = requests.get(url, headers=headers)

Some servers block bots by checking the User-Agent.

robots.txt

text

https://example.com/robots.txt

This file specifies which paths are allowed and disallowed for crawling. It is polite and legally prudent to first check the site's crawling policy.

Limitations of Crawling

BeautifulSoup can only process static HTML. Content that is dynamically loaded with JavaScript (SPAs, infinite scrolling, etc.) is not included in the HTML and cannot be extracted.

Situation	Tool
Static HTML	requests + BeautifulSoup
Dynamic loading with JavaScript	Selenium, Playwright
API available	Use requests to call the API directly (most efficient)

If there is an API, use the API instead of crawling. This allows you to receive structured data accurately and reduces the load on the server.

Summary of the Core Flow

text

1. requests.get(url) → HTML text
2. BeautifulSoup(html) → Parsed tree
3. soup.select("CSS selector") → Desired element
4. element.text / element["attr"] → Extract data

Error Handling

In real-world crawling, various errors can occur:

python

import requests
from bs4 import BeautifulSoup

def safe_fetch(url, retries=3):
    for attempt in range(retries):
        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            return response.text
        except requests.exceptions.Timeout:
            print(f"Timeout ({attempt + 1}/{retries})")
        except requests.exceptions.HTTPError as e:
            print(f"HTTP error: {e}")
            return None
        except requests.exceptions.ConnectionError:
            print(f"Connection failed ({attempt + 1}/{retries})")
    return None

If you do not set a timeout, the program will wait indefinitely for a server that does not respond. raise_for_status() raises an exception for 4xx/5xx responses.

Crawling Multiple Pages

python

import time

base_url = "https://example.com/articles?page="
all_titles = []

for page in range(1, 11):
    html = safe_fetch(f"{base_url}{page}")
    if html is None:
        continue
    
    soup = BeautifulSoup(html, "html.parser")
    titles = soup.select("h2.article-title")
    all_titles.extend([t.text.strip() for t in titles])
    
    print(f"Page {page}: {len(titles)} items collected")
    time.sleep(1)

print(f"Total of {len(all_titles)} items collected")

Saving Crawled Data

python

import json
import csv

# Save as JSON
with open("articles.json", "w", encoding="utf-8") as f:
    json.dump(all_titles, f, ensure_ascii=False, indent=2)

# Save as CSV
with open("articles.csv", "w", encoding="utf-8", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["ID", "Title"])
    for i, title in enumerate(all_titles, 1):
        writer.writerow([i, title])

Save the collected data in JSON or CSV format so that it can be easily analyzed with pandas.

Extracting Links — href Attribute

python

# Extract all links on the page
links = soup.select("a[href]")
for link in links:
    url = link["href"]
    text = link.text.strip()
    print(f"{text}: {url}")

# Only links matching a specific pattern
article_links = [
    a["href"] for a in soup.select("a[href]")
    if "/article/" in a.get("href", "")
]

Crawling is the first step in "getting data." Data collection is the starting point for everything, from data analysis and machine learning to monitoring.

BioPlayground

Web Crawling Basics — BeautifulSoup

Web Crawling Basics — BeautifulSoup

After completing this topic

What is Web Crawling?

Step 1: Fetching HTML (requests)

Step 2: Parsing HTML (BeautifulSoup)

Step 3: Extracting Data

Finding by Tag

Finding by CSS Selector

Practical Example: Extracting Table Data

Points to Note

Request Interval

Setting User-Agent

robots.txt

Limitations of Crawling

Summary of the Core Flow

Error Handling

Crawling Multiple Pages

Saving Crawled Data

Extracting Links — href Attribute

💬 Questions & Comments