Web Scraping Explained: Lawful, Careful Data Collection

Written by Danish Mustafa · Reviewed by Ahmad Jajja · June 15, 2026

Imagine a shop owner who wants to compare the prices of fifty products across ten stores. Opening each page by hand, copying each price into a spreadsheet, and repeating the work every week would take hours and invite typing errors. A program that fetches those pages and pulls out just the product names and prices can do the same job in minutes, and it can repeat the job the same way every time. That program based approach is the core idea behind collecting data from web pages, and it is useful in research, price tracking, market study, and monitoring of public information, when it is done with care and with respect for the sites involved.

At its simplest, the process has four steps. First, the program requests a page, much as a browser does, and receives the page content. Second, it parses the content, which means it reads the page structure and finds the pieces it needs, such as a title inside a heading or a price inside a labelled field. Third, it cleans those pieces, removing extra spaces, symbols, and duplicate rows. Fourth, it stores the result in a structured form, such as a table or a database, where the data can be searched, compared, and charted. Each step can fail, and good practice means handling those failures calmly, rather than hammering a site with retries when a page is missing or a server is busy.

This guide explains the technique in plain language, then spends equal time on the limits that matter: robots.txt, which is a file a site uses to tell automated visitors which paths they may fetch, terms of service, rate limits, and rules for personal data. It also explains when you should not scrape at all, and when an official feed or API, which is a structured data service a site offers on purpose, is the better choice. Before you build anything, check Google's guide to robots.txt and the robots.txt resource site, because those two sources describe the standard that most sites follow.

What the Technique Actually Does

A web page is not just text on a screen. Behind the visible page sits HTML, which is the markup language that structures a page into headings, paragraphs, lists, tables, and links. A scraping program reads that structure and selects elements by their tags, labels, or class names, which are markers authors add to style and organize parts of a page. For example, a product list may mark every price with the same label, so the program can find all prices with one rule. When pages follow a steady template, one rule works across thousands of pages, which is why the technique scales so well for catalogs, listings, and directories.

Some pages load their content with JavaScript, which is code that runs in the browser and fills in content after the first load. A simple fetcher that reads only the first response may see an empty page, because the content had not arrived yet. Tools that drive a real browser can wait for that content, but they cost more time and computing power, so use them only when the simpler approach truly fails and the site permits automated access. In many cases, the site already offers the same data through a feed, a download, or a search page that returns plain HTML, and finding that simpler path saves both time and server load. Our guide on how companies connect their systems explains why an official data channel usually beats a fragile page based workaround.

Common Uses, and Uses to Avoid

Legitimate uses start from public information and a clear purpose. A researcher may gather public job listings to study hiring trends across cities. A small business may track its own products' public prices on marketplaces where it sells. A journalist may monitor public government notices for changes over time. A team may archive its own published pages for quality checks. In each case, the data is public, the purpose is stated, the load on the site is small, and the result does not republish someone else's creative work as if it were new. Those features together make a project defensible, even though they do not remove every legal question.

Uses to avoid are equally clear. Do not collect personal data about private people, such as private phone numbers, private email addresses, or location trails, even when a page shows them. Do not copy full articles, books, photos, or reviews and republish them, because that is a copyright problem even when the copy was made by a program. Do not scrape a site to clone its product, undercut its service by stealing its catalog wholesale, or build a competing database from its hard work without permission. Do not use scraping to gather login protected content, to dodge a paywall, or to defeat a technical barrier a site put in place on purpose. If the value of your project comes mainly from taking what someone else built and reselling it, find a licensed source instead.

Robots.txt, Terms of Service, and Permission

Robots.txt is a small text file a site places at its root, such as example.com/robots.txt, to tell automated visitors which paths they may fetch and which they must leave alone. It uses simple rules: a user agent line names the visitor, a disallow line names a path to avoid, and a crawl delay line, when present, asks visitors to wait between requests. Following robots.txt is both polite and practical, because sites watch for visitors that ignore it, and ignoring it can lead to a block that ends the project anyway. Always fetch and read the file before your first real request, and recheck it when a project runs for months, because site rules change.

Terms of service are the rules a site publishes for using it, and they often contain a section on automated access. Some sites allow it for public pages with limits. Some allow it only with written permission. Some ban it outright. Read that section before you write code, not after a block or a complaint, and treat a ban as a stop sign, not as a puzzle to solve. When the terms are unclear, ask the site owner for permission in writing, explain exactly what you will collect, how often, and how you will use it, and keep the reply with the project records. Permission turns a gray area into a clear one, and it often leads to a better data source, because owners who say yes may point you to a feed you missed.

Permission also matters for data that is not yours to share. Even public pages can contain names, photos, and comments from real people, and privacy laws in several regions limit how such data may be collected, stored, and shared. Collect the smallest set of fields that answers your question, avoid fields that identify private people, and delete data you no longer need. If your project touches European users, check the General Data Protection Regulation, which is the European Union privacy law, and if it touches California users, check the California Consumer Privacy Act, because both laws can apply even when your team sits elsewhere. When in doubt, ask a lawyer before launch, because a short legal review costs far less than a complaint after publication.

Polite Crawling: Rate Limits, Caching, and Retries

A polite crawler acts like a considerate guest. It waits between requests, often one to several seconds, and it slows down further when the server responds slowly or returns errors. It identifies itself with a clear user agent string, which is the name a program sends to say what it is, including a contact address, so the site owner can reach a human if there is a problem. It fetches each page once and caches the result, which means it saves a copy for reuse, instead of fetching the same page again for every small test. These habits protect small sites in particular, because a small site on shared hosting can be knocked over by a careless program that sends hundreds of requests in a minute.

Rate limits also protect your own project. A server that returns a 429 status code is telling you to slow down, and the right response is to wait, back off, and try later, not to send more requests from more addresses to get around the limit. Treat repeated 403 responses, which mean forbidden, and blocks as a signal to stop and review the rules, not as a challenge to evade. Keep a log of every request, including the time, the URL, and the response, so you can show what you did if a site owner asks. A clean log, a slow pace, and a willingness to stop are the strongest evidence that a project was run in good faith.

Tools for Public Pages, Used With Permission

For pages that return plain HTML, a parser library is usually enough. In Python, which is a popular programming language for data work, libraries such as BeautifulSoup and Scrapy help a program read page structure and select elements by their labels. BeautifulSoup suits small jobs and learning, because it focuses on parsing one page at a time. Scrapy suits larger jobs, because it adds request scheduling, polite delays, and pipelines for cleaning and storing results. Both tools are general purpose: they fetch and parse pages, and they do not exist to break logins, paywalls, or anti-bot protections, so this guide does not recommend any tool whose purpose is to bypass those protections.

Choose the tool that matches the job and the site's rules. If the site offers an API, use it first, because an API returns structured data by design and usually carries clear terms. If the site offers a data download or a feed, use that next. Only when neither exists, and the rules allow automated access, should you parse pages directly. Test on a handful of pages first, check the output by hand, and fix your selection rules before scaling to thousands of pages, because a wrong rule repeated at scale creates a large pile of bad data that looks convincing. Our guide to coding assistants can help you write the parsing code faster, and our AI and machine learning development services cover the cleaning and modelling steps that follow collection. For search data in particular, checking search engines other than Google can reveal an official route before scraping a results page.

Cleaning, Storing, and Checking the Data

Raw scraped data is rarely ready to use. Prices arrive with currency symbols and commas, dates arrive in several formats, and the same product may appear under slightly different names. Cleaning means turning those varied forms into one consistent form: numbers become numbers, dates become one format, and duplicate rows are removed or merged with a stated rule. Keep the original values beside the cleaned values during development, so you can trace a strange result back to the page it came from. Record the source URL and the fetch date for each row, because a dataset without sources cannot be checked, corrected, or trusted by anyone else.

Quality checks should run before any analysis. Count rows per source and per day, and investigate sudden jumps or drops, because they often mean a page template changed or a block began. Check a random sample by hand against the live pages, and check that key fields are present for nearly every row, because a field that is empty half the time points to a broken rule. Store the data in a form that suits its size, such as a spreadsheet for a small project or a database for a large one, and restrict access to the people who need it. A dataset that is cleaned, sourced, and checked will support honest conclusions, while a dataset that skips those steps will support confident errors.

A Safe Workflow From Idea to Result

Handling Change When Pages Move

Pages change, and a scraping project must expect that change rather than be surprised by it. A site may redesign its layout, rename a label, move a price into a new field, or split one page into several pages. When that happens, a selection rule that worked for months can suddenly return empty fields or, worse, the wrong fields with no warning. The defence is monitoring: count rows every run, count empty fields every run, and alert a human when either number moves sharply. A run that returns half the usual rows is not a quiet success. It is a signal to pause, inspect a few pages by hand, and fix the rule before the bad data spreads into reports or decisions.

Design rules that survive small changes. Prefer stable markers, such as a labelled field or a data attribute that describes the content, over fragile markers such as the exact position of an element on a page. Position based rules break when a site adds one banner or one promotion box, while label based rules often survive a redesign. Keep each rule small and named, so a fix touches one rule instead of a tangled set of guesses. Version your rules beside your code, and keep a small set of saved example pages for testing, taken from pages you are allowed to fetch, so you can check a new rule against old examples before it runs live. These habits turn a brittle script into a maintainable tool, and they reduce the temptation to fetch pages more often just to check whether a fix worked.

Change also comes from the site owner, not only from layout. A site may add a stricter robots.txt rule, publish a new terms section, or launch an official API that replaces page based access. Treat each of those events as a prompt to recheck permission, not as an obstacle to route around. Switching to the official API at that point often improves the project, because structured data needs less cleaning and carries clearer terms. If the new terms ban your use, stop that source and find another way to answer the question. A project that adapts to owners' choices keeps its access and its reputation, while a project that fights those choices loses both, usually at the worst moment.

That workflow also tells you when to walk away. Walk away when a site bans automated access, when the data sits behind a login or a paywall, when the project needs personal data about private people, or when the main value comes from republishing someone else's content. Walking away is not a failed project. It is a project that protected its team from blocks, complaints, and legal risk, and it leaves room to find a licensed source, a partner, or a different question that public data can answer. Teams that treat "do not scrape this" as a normal outcome run more projects over time, because they keep their reputation and their access intact.

Conclusion

Collecting data from web pages works best when it starts from a clear purpose, a public source, and respect for the site that hosts the data. Check robots.txt and terms, prefer an official feed or API, crawl slowly, collect only what you need, and stop when asked. Follow that discipline, and you can build datasets that are useful, checkable, and defensible, without harming sites, copying creative work, or collecting personal data you should not hold.

Frequently Asked Questions

Relevant blogs

Back to all blog guides · Back to Software Development guides · Explore AI and ML Development Services