Web Scraping Legality: What Business Owners Need to Know

Public vs Non-Public Data: The Foundation of Is Web Scraping Legal

The clearest rule in web scraping is the distinction between public and non-public data. Information visible to anyone without logging in—product listings, business directories, real estate ads, news articles—is generally fair game. Courts in both the US and UK have repeatedly held that accessing publicly available websites does not constitute unauthorised access under computer misuse laws like the Computer Fraud and Abuse Act (CFAA) in the United States or the Computer Misuse Act 1990 in the UK.

Non-public data is different. If you need to log in, bypass a paywall, or use someone else's credentials to reach the information, you cross into unauthorised access territory. The CFAA criminalises exceeding authorised access to computer systems, and UK law similarly prohibits accessing data you are not entitled to see. The hiQ Labs v. LinkedIn case centred on this issue: LinkedIn argued that scraping violated the CFAA even after hiQ was told to stop, but the Ninth Circuit initially ruled that public profiles were not protected. The case was ultimately settled, leaving some ambiguity, but the principle remains—scraping publicly visible data is materially different from accessing gated content.

For business use, stick to data that any visitor can see without signing in. If your project requires account access, you must own that account or have explicit permission from the account holder.

Personal Data and Privacy Law

Collecting personal data—names, email addresses, phone numbers, IP addresses, location data—triggers privacy regulations. In the UK, the UK GDPR and Data Protection Act 2018 require a lawful basis for processing personal data, transparency about how you will use it, and respect for individuals' rights. In the United States, California's CCPA and CPRA impose similar requirements for California residents, and other states are following suit with their own laws.

The lawful basis matters. If you are scraping business contact details from a public directory for B2B marketing, you might rely on legitimate interests (UK GDPR) or claim the data falls outside consumer privacy laws (US). If you are collecting consumer profiles, reviews with names, or social media activity, you need to think harder about consent, purpose limitation, and data subject rights. The European Data Protection Board has made clear that publicly available does not mean exempt from data protection law—you still need a lawful basis and must respect individuals' reasonable expectations.

Practically, many businesses avoid personal data altogether. Instead of scraping user reviews with names attached, scrape aggregate ratings and anonymous text. Instead of collecting individual social media profiles, collect public posts without tying them to identifiable people unless you have a clear legal path and a privacy policy to match. If your project requires personal data, consult a data protection lawyer before you start.

Terms of Service: Contract vs Technical Access

Most websites publish terms of service that prohibit scraping, automated access, or commercial use of their data. The legal weight of these terms is a frequent point of confusion. In the United States, violating terms of service does not automatically make scraping a criminal act under the CFAA—courts have held that a terms-of-service violation alone is not the same as unauthorised access to a computer system.

However, terms of service remain contracts, and breaching them can lead to civil consequences. A website owner can send a cease-and-desist letter, seek an injunction, or sue for breach of contract if they can show damages. In the UK, contract law applies similarly—if you explicitly agree to terms that forbid scraping, you may be liable for breach, though enforcement is rare unless the website can demonstrate actual harm.

How to Approach Terms of Service

• Read the terms. If scraping is explicitly prohibited and you are planning a commercial project, weigh the risk. Large platforms with aggressive legal teams are likelier to enforce their terms than small sites.
• Check for an API first. Many platforms offer official APIs that give you structured data without contractual ambiguity.
• Stay polite and technical. Even if terms forbid scraping, websites often tolerate low-volume, respectful bots that do not disrupt service. The line between tolerated and litigated often comes down to scale and impact.
• Document your process. If challenged, showing that you collected only public data, respected robots.txt, and kept request rates reasonable strengthens your position.

For grey-area projects—scraping a major marketplace or real estate platform for price monitoring—the practical advice is to proceed cautiously, use a reputable scraping service that understands the boundaries, and be prepared to stop if you receive legal pushback.

Copyright and Database Rights

Copyright protects original creative works—text, images, videos, software code. Scraping a website does not give you ownership of its content. If you republish scraped articles, product descriptions, or photographs without permission, you risk copyright infringement. In the UK and EU, database rights add another layer: the Database Directive grants sui generis protection to databases where substantial investment has gone into obtaining, verifying, or presenting the data.

The United States does not have database rights, but copyright still applies to original compilations and creative content. Facts themselves are not copyrightable—prices, addresses, product specifications—but the way they are expressed or arranged might be. Scraping raw factual data for your own internal use (price comparison, market research, inventory monitoring) generally falls under fair use or does not implicate copyright at all. Republishing that data in a way that competes directly with the original source is riskier.

Minimising Copyright Risk

1. Extract facts, not creative content. Scrape prices, specifications, and availability, not marketing copy or professional photographs.
2. Transform the data. If you aggregate, analyse, or reformat scraped data into your own product, you add transformative value that supports a fair use argument.
3. Attribute where sensible. If you display scraped data publicly, link back to the source.
4. Avoid bulk republication. Mirroring an entire website or database is the clearest infringement. Use scraped data to inform decisions, not to replace the original.

For commercial projects that will republish or redistribute scraped content—comparison sites, aggregators, AI training datasets—spend the money on a legal review.

Technical Impact: Respect the Infrastructure

Even when you have a legal right to scrape, the technical manner matters. Overloading a website with requests can constitute a denial-of-service attack or trespass to chattels (a US common-law doctrine allowing property owners to sue for interference with their systems). UK law has no direct equivalent, but aggressive scraping that causes measurable harm—downtime, increased hosting costs, degraded user experience—can support claims for damages or an injunction.

Rate limiting is both a courtesy and a legal safeguard. A well-designed scraper mimics human behaviour: it waits between requests, respects robots.txt directives, identifies itself with a user-agent string, and backs off if the site returns errors. Websites deploy anti-bot measures not just to enforce terms of service but to protect infrastructure.

Robots.txt is not legally binding, but ignoring it weakens your position if a dispute arises. Courts have considered compliance with robots.txt as evidence of good faith, and violating it can support a claim that you knowingly exceeded authorised access. For business projects, configure your scrapers to honour robots.txt unless you have a specific reason—and legal advice—to override it.

Practical Compliance Checklist

Every scraping project should begin with a compliance review. The following checklist covers the main legal and technical considerations for businesses in the US and UK. It is not a substitute for legal advice, but it gives you a framework to identify risks before you commit resources.

Before You Start

• Confirm the data is public. Can any visitor see it without logging in or paying? If yes, proceed. If no, stop or get permission.
• Check for an official API. Many platforms provide structured data access that avoids contractual ambiguity.
• Read the terms of service. Note any explicit prohibitions on scraping, automated access, or commercial use. Weigh the enforcement risk based on the platform's history and your use case.
• Identify personal data. Will you collect names, emails, phone numbers, or other identifiers? If yes, determine your lawful basis under UK GDPR or CCPA.
• Review copyright and database rights. Are you extracting creative content (text, images) or factual data (prices, specs)? Plan to use facts, not copy, and add transformative value if possible.

During the Project

• Respect robots.txt. Configure your scraper to honour disallow directives unless you have a legal reason to override them.
• Limit request rates. Space requests to avoid overloading the target site. One request per second is a good starting point for small sites.
• Use a clear user-agent. Identify your scraper with a descriptive user-agent string and provide contact information.
• Monitor for blocks and errors. If you receive 403, 429, or captcha challenges, slow down or stop.
• Store data securely. If you collect personal data, encrypt it in transit and at rest, limit access, and comply with data retention rules.

After Collection

• Use data for the stated purpose. Do not repurpose personal data without a fresh legal basis.
• Attribute sources where appropriate. If you display scraped data publicly, link back to the original site.
• Respond to takedown requests. If a website owner asks you to stop or delete data, take the request seriously.
• Review periodically. Laws change, terms of service update, and enforcement priorities shift. Revisit your compliance review annually.

When to Seek Legal Advice

Some projects fall squarely within safe boundaries—scraping public product listings for internal price comparison, collecting real estate ads for market analysis, or gathering customer reviews for sentiment research. Others are ambiguous or high-risk. You should consult a lawyer if your project involves any of the following:

• Scraping a platform that has sent cease-and-desist letters or sued scrapers in the past.
• Collecting personal data at scale, especially if you plan to sell it, use it for profiling, or combine it with other datasets.
• Republishing or redistributing scraped content in a way that competes with the original source.
• Accessing data behind login walls, paywalls, or technical protections, even if you have an account.
• Operating in a regulated industry (finance, healthcare, legal services) where data handling is subject to sector-specific rules.

A lawyer cannot make an illegal project legal, but they can help you structure a defensible approach, draft terms and privacy notices, and advise on risk mitigation. For high-value projects—building a SaaS platform on scraped data, launching a comparison site, or creating a commercial dataset—the cost of legal advice is modest compared to the cost of litigation or regulatory fines.

Working with a Scraping Service

Many businesses choose to outsource data collection rather than build scrapers in-house. A professional service brings technical expertise, handles anti-bot protection, and understands the legal boundaries. When evaluating a provider, ask about their compliance practices. Do they collect only public data? Do they respect robots.txt and rate limits? Do they avoid personal data unless you provide a lawful basis?

Kodrana maintains a catalog of ready scrapers for common platforms—marketplaces, real estate sites, job boards, business directories—and builds custom scrapers for new sources within 2-5 business days. We collect publicly available data only, never log into accounts we do not own, and configure every scraper to respect technical and legal limits. Before starting any project, we review the target site and confirm feasibility. For new clients, we provide a free sample of 50-100 records so you can verify data quality and compliance before committing. If you are unsure whether your project is viable, send us the site and the fields you need—we will check the protection level and advise on the legal and technical path forward.