What Protected Websites Mean for a Data Collection Project

Why websites block scrapers

Website operators block or limit automated requests for valid commercial and operational reasons. An online retailer may need to protect stock availability and checkout performance. A job board may want to prevent wholesale copying of its listings. A publisher may be defending paid content, while a directory may be trying to stop its database from being replicated and resold. The fact that information can be viewed in a browser does not give a project unlimited permission to retrieve or reuse it.

Protection is also about stability. A person might open a few product pages in one session; an unplanned script can request thousands of pages in minutes, repeatedly reload the same content, or follow filter combinations that create large numbers of server requests. Even a small business website can struggle with that pattern. This is why a data specification should include the minimum fields, locations, categories and update frequency needed for the actual decision.

The usual protections and what they signal

• Rate limits restrict how many requests a visitor or application can make during a period. They usually signal that the source expects measured, predictable use or prefers an official data interface.
• CAPTCHAs ask a visitor to complete a challenge when traffic appears unusual. They are intended to separate human interaction from automated activity and should not be treated as an obstacle to work around.
• Cloudflare can provide content delivery, security and traffic management. Depending on the site’s settings, it may challenge, block or slow requests that do not resemble permitted normal access.
• DataDome and comparable anti bot protection products analyse traffic patterns and other signals to identify suspected automation or abuse. Their presence means the project should reassess access permissions and alternatives, not escalate collection pressure.
• JavaScript-rendered pages may deliver little useful content in the initial HTML. This is often a normal product-design choice, rather than a ban on collection, but it changes the technical effort needed to reliably read public page content.

For pricing teams, this distinction matters. Monitoring a limited public product catalogue with an agreed refresh schedule is a different project from attempting to duplicate an entire marketplace. For practical planning, see the considerations behind competitor price monitoring and the broader overview of API versus web scraping.

Scraping Cloudflare protected sites: assess access before building

When a site uses Cloudflare, do not assume either that collection is impossible or that it is acceptable. Cloudflare is widely used for many functions, and the relevant question is whether the intended access is authorised, proportionate and technically sustainable. Start with the website’s terms, robots guidance where available, API documentation, partner programmes and data licensing options. If the site offers an official API, feed, export or affiliate data source that serves the business need, that is normally the preferred route.

1. Write a narrow data brief: the public pages or categories, fields required, geography, volume, refresh interval, delivery format and intended business use.
2. Check official routes first, including APIs, downloadable feeds, supplier integrations, licensing and partnerships. Record any usage limits and attribution or storage requirements.
3. Review terms of service and applicable legal constraints. Where the source, reuse rights or jurisdiction creates uncertainty, ask a qualified lawyer about the specific use case.
4. Run a limited feasibility check rather than committing to a large extraction. Confirm that the needed public fields are available and that collection can be kept gentle.
5. Set stop conditions. Repeated blocks, explicit restrictions, signs of strain on the source or a request from the operator should trigger review, not more aggressive automation.

A useful feasibility test answers business questions as well as technical ones: Is an API complete enough? Is daily change genuinely necessary, or would weekly collection work? Can the project collect only changed items? Does the site expose a public search result but withhold key fields on each detail page? Those answers determine cost and whether the project should proceed.

For recurring source work, a custom service should document the data contract and changes in the target site. Kodrana’s data extraction service focuses on publicly available data, with the collection scope defined before delivery rather than relying on an open-ended crawl.

Rate limits, caching and polite collection design

Rate limits are not merely an inconvenience; they are a design input. A project that must refresh 20,000 listings every hour may be unsuitable for a public website even if the data is visible to visitors. By contrast, a smaller catalogue refreshed once a week, with duplicate requests removed, may meet the business need while placing far less load on the operator.

Reduce demand before asking for more data

• Collect only fields that support a defined decision. A price-monitoring table may need product name, URL, price, currency, availability and capture time, not every image, review or related-product widget.
• Use an update schedule based on volatility. Fast-moving inventory may justify more frequent checks; stable property or directory records may not.
• Cache previously retrieved public records and request updates only where there is a credible reason to expect a change. Avoid repeatedly fetching unchanged pages.
• Deduplicate URLs, products and listings before collection. Variant pages and tracking parameters can inflate request counts without adding useful records.
• Build validation into the pipeline so a malformed page, missing price or changed page layout is flagged rather than silently written into a reporting table.
• Schedule predictable, measured collection windows and avoid sending bursts that could affect a source’s normal users.

Caching also improves the economics of a project. It reduces source traffic, shortens processing time and helps analysts compare a current record with a prior snapshot. A well-designed feed keeps capture time, source URL, unique source identifier where publicly available, and data-quality status alongside the business fields. This creates an audit trail when a price or listing later changes.

Teams collecting marketplace data can often reduce the scope substantially by agreeing which sellers, brands or categories matter first. The marketplace scraper catalogue can help identify existing source coverage, while a bespoke plan is more suitable where the source structure or output requirements are specific.

CAPTCHAs and anti bot protection are project boundaries

A CAPTCHA is an explicit signal that a site wants additional verification before allowing the next action. Similarly, a block page, challenge page or repeated access denial should be treated as a boundary. A responsible provider does not present methods for defeating these measures as a standard delivery feature. Instead, pause and consider whether an official source, permission from the operator, a smaller scope or a different dataset can achieve the business outcome.

DataDome and other anti bot protection services may respond differently depending on traffic volume, route, time and the site’s own policies. That variability makes promises such as “permanent access” unrealistic. It also explains why an honest project estimate includes discovery, monitoring and a plan for source changes. The objective is dependable, permitted access to the necessary public data, not a contest with a security system.

For example, an e-commerce manager comparing a handful of public competitor prices could use supplier feeds, brand-authorised data, public price-comparison sources, or a reduced page set. A recruitment analyst may be better served by a job board’s partnership route, authorised feed, or a combination of public employer career pages. For category-specific scope planning, see job data sources and public review sources.

JavaScript rendering changes the build, not the rules

Many modern sites load product cards, prices, availability, maps or filters after the initial page opens. JavaScript rendering can make simple HTML extraction incomplete because the visible browser page is assembled from later requests and interface state. It may also make page layouts more sensitive to design changes. None of this changes the need to use public data only and respect the site’s access conditions.

The practical question is whether the visible public page contains the needed fields and whether the project can retrieve them at a low, stable volume through an allowed method. A developer may need to account for loading states, pagination, regional settings, consent banners and pages that return different content when an item is out of stock. These are data-quality issues, not reasons to bypass controls.

• Define a record schema before collection: for example, listing title, public URL, price, location, availability, source identifier and observed-at timestamp.
• Specify acceptable gaps. If a public page has no phone number, seller identity or precise address, do not attempt to infer or obtain it from restricted areas.
• Plan for layout changes. Use alerts for missing key fields and establish a review process before incomplete records reach dashboards or CRM workflows.
• Keep source data separate from calculated fields such as price differences, availability labels or lead scores, so results can be checked later.

Where public listing feeds are appropriate, real estate teams should account for quickly changing availability and duplicate adverts. The real estate scraper category is a useful starting point for identifying relevant public sources and defining a narrower, cleaner dataset.

Legal and data-use checks for UK, US and EU projects

Technical feasibility does not settle legality. A project involving UK personal data must consider UK GDPR; EU personal data can bring GDPR obligations; and California activities may require review under the CCPA/CPRA. Personal data can include information that identifies or can reasonably be linked to a person, depending on context. A business should not assume that public visibility removes all privacy obligations.

In the United States, the Computer Fraud and Abuse Act (CFAA) is one reason to take access restrictions seriously. In the UK, the Computer Misuse Act 1990 is relevant to unauthorised access concerns. Copyright can apply to page content, images and editorial material, while the EU database right may protect substantial investment in obtaining, verifying or presenting database contents. Contract terms, intellectual property rights, privacy law and the jurisdiction of the source can all affect the answer.

For a specific source and intended use, obtain advice from a qualified lawyer. That is particularly sensible when data will be republished, used to train models, combined with customer data, transferred across borders, or used for credit, employment, housing or other consequential decisions. Legal review should cover both collection and downstream storage, retention, sharing and deletion.

A responsible operating checklist

• Use official APIs, partner feeds and written permission where available and suitable.
• Limit collection to clearly public business data required for a documented purpose.
• Do not access accounts belonging to other people, restricted pages, paywalled material or information behind logins without proper authorisation.
• Do not collect personal data unless there is a lawful, reviewed basis and appropriate privacy controls. Apply minimisation, retention limits and access controls.
• Respect terms, stated restrictions and reasonable technical boundaries. Stop and reassess if access is denied or challenged.
• Document source, collection date, permitted use, transformations and delivery recipients.

Set the right delivery model and maintenance expectations

Before commissioning work, agree whether the need is a one-off export, an ongoing monitoring feed or a broader business workflow. A one-off file may be enough for market research. A recurring feed needs source-change monitoring, validation rules, data retention and an owner on the client side who can decide what to do with missing or changed records. Data is most useful when it arrives in the system where someone can act on it.

Kodrana can deliver public data as CSV, Excel, JSON, Google Sheets, an API or webhook, or directly into a CRM or database. The studio also builds automation and integrations where data needs to trigger a workflow, such as routing a stock change to a reporting system. See business automation services for workflow-oriented projects and the guide to scraping tools versus custom work when deciding between an off-the-shelf tool and a managed build.

For a practical starting point, Kodrana can review a target site and required public fields, check the protection level, and provide a free sample of 50-100 records before payment. This is a useful way to validate coverage, formatting and lawful scope before requesting a quote for a one-off export or ongoing feed.