How to Scrape the Web with Python: A Practical Guide with requests and BeautifulSoup

• 27 June 2026
• by: Ravi Subramanian
• in: Web hosting
• Tags: beautifulsoup, Python, web scraping python
• note: no comments

Web scraping is the practice of extracting data from web pages programmatically instead of copying it by hand. You write code that fetches a page, reads its HTML, picks out the pieces you care about — prices, headlines, listings, tables — and saves them somewhere structured like a CSV or a database. When you want to scrape web Python is almost always the right tool, because its ecosystem is mature, readable, and battle-tested for exactly this kind of work.

This guide is developer-to-developer. We will build a working scraper with `requests` and `BeautifulSoup`, handle the real-world problems that break naive scrapers, and talk about where to actually run these jobs once they work. But first, the part most tutorials skip.

Key Takeaways
• Web scraping = fetching HTML and parsing it. `requests` fetches the page, `BeautifulSoup` parses the HTML so you can select elements.
• Legality and ethics come first. Respect `robots.txt`, terms of service, and rate limits. Scrape public data, don’t hammer servers, identify yourself with a `User-Agent`.
• The decisive question is static vs. dynamic. If the data lives in the page’s initial HTML, `requests` + `BeautifulSoup` works. If JavaScript builds it client-side, you need Selenium/Playwright or the site’s API.
• `requests` + `BeautifulSoup` covers most jobs; Scrapy scales to large crawls; Selenium/Playwright handle JS-rendered pages.
• Run scrapers on a server. A VPS with cron turns a one-off script into a scheduled, reliable data pipeline.

Is web scraping legal and ethical?

Before you write a line of code, understand the boundaries. Web scraping is not inherently illegal, but how you do it matters, and being careless can get your IP blocked, your account banned, or worse.

A few principles to scrape responsibly:

• Check `robots.txt`. Most sites publish rules at `https://example.com/robots.txt` describing which paths crawlers may access. It is not a legal contract, but it signals the site owner’s intent — honor it.
• Read the terms of service. Some sites explicitly prohibit automated access. Scraping anyway can violate their terms.
• Prefer public data. Data behind a login, paywall, or that includes personal information carries far more legal and ethical risk. Stick to publicly visible, non-personal data when you can.
• Don’t overload the server. A scraper that fires hundreds of requests per second looks like a denial-of-service attack. Add delays, limit concurrency, and scrape during off-peak hours.
• Identify yourself. Set a descriptive `User-Agent`. Some scrapers even include a contact URL so admins can reach you instead of just blocking you.

The short version: be a good citizen. Take only what you need, at a pace the server can absorb, and look for an official API first — it is almost always the better path.

What is the Python web scraping toolkit?

Python gives you a small set of focused tools. You rarely need all of them at once; you pick based on the page.

Library	Use it when
`requests`	You need to fetch a page or call an API over HTTP. The standard for sending GET/POST requests.
`BeautifulSoup` (bs4)	You have raw HTML and need to parse it — find tags, read attributes, extract text.
`lxml`	You want a fast parser backend for BeautifulSoup, or you prefer XPath selectors.
`Scrapy`	You’re building a large crawler: many pages, concurrency, pipelines, retries, and built-in throttling.
`Selenium` / `Playwright`	The page renders content with JavaScript and you need a real browser to execute it.

For the majority of scraping tasks, `requests` to fetch plus `BeautifulSoup` to parse is the whole toolkit. We’ll focus there, then cover when to reach for the heavier options.

Install the two essentials:

“`bash pip install requests beautifulsoup4 “`

How do you build a basic scraper with requests and BeautifulSoup?

Let’s build a scraper step by step. The pattern is always the same: fetch, parse, select, extract, save.

Fetching a page with requests

“`python import requests

url = “https://example.com/products” response = requests.get(url, timeout=10)

response.raise_for_status() # raises an error on 4xx/5xx

html = response.text print(response.status_code) # 200 means success “`

`requests.get()` returns a `Response` object. `response.text` is the raw HTML as a string, `response.status_code` tells you whether the request succeeded, and `raise_for_status()` fails loudly on errors instead of letting you parse an error page by accident.

Parsing HTML with BeautifulSoup

“`python from bs4 import BeautifulSoup

soup = BeautifulSoup(html, “html.parser”)

print(soup.title.text) # the