Web Scraping Explained: From Website Pages to Usable Data

What is web scraping, and what is data extraction?

People often use web scraping and data extraction interchangeably, but there is a useful distinction. Web scraping is the process of reading information from website pages or public web interfaces. Data extraction is the broader business task: collecting, cleaning, structuring and delivering information from a source so it can be used elsewhere. A scraper is the software that performs the collection according to rules you define.

For example, a retailer may need a daily table of competitor product names, stock status, listed prices, URLs and collection dates. The scraper gathers those public fields; the extraction workflow then matches products, standardises currencies or pack sizes, removes duplicates and sends the result to a spreadsheet, database or pricing system. A public marketplace listing, property listing, job advert or review page can be a source, provided the project is lawful and respects the site’s rules and technical limits.

A scraper is not necessarily the right answer when a source offers a reliable API with the fields and usage rights you need. APIs are designed for system-to-system access, whereas web scraping reads publicly presented website data. The practical differences are covered in API versus web scraping.

What scraping is not

• It is not permission to access private dashboards, paid content, login-only pages or another person’s account.
• It is not a way to collect personal data without a lawful basis, proper notices and an appropriate compliance review.
• It is not a one-time technical task if the source changes regularly. Sites alter page layouts, product structures and availability.
• It is not the same as copying an entire website. A well-scoped project collects only the defined public fields needed for a business purpose.

How a scraper works step by step

A dependable data collection project begins with a specification rather than code. The business team should identify the sources, exact fields, geographical coverage, filters, frequency, volume and intended destination. “Get competitor prices” is too broad; “collect public listed prices, seller names and stock labels for 2,000 matched SKUs every morning” is actionable.

1. Define the data model. List each column, its format and an example value. Include a source URL and collection timestamp so the output can be audited.
2. Identify public pages that contain the required information. This can include search-result pages, category listings, individual item pages or public directories.
3. Fetch the permitted public pages at a measured rate. The collection process should minimise unnecessary requests and avoid putting undue load on the source site.
4. Read the relevant page elements or public data responses and extract the specified fields. A scraper may follow pagination and individual listing links where that is within the approved scope.
5. Validate and clean the results. This commonly means normalising dates, removing duplicates, checking required fields and flagging unusual values.
6. Match and enrich records when needed. For price monitoring, the task may involve matching the same product across several retailers by SKU, model number or another reliable identifier.
7. Export or deliver the dataset. The process can run once for a research project or recur on a daily, weekly or other agreed schedule.
8. Monitor the source and output quality. If a site changes its public page structure, the extraction rules may need maintenance.

The collection method depends on the website. Some pages contain the relevant text directly in the initial HTML; others load public listings after the page opens. The key business point is not the technical mechanism but whether the final dataset remains complete, traceable and consistently structured. For recurring projects, agree on data-quality checks before collection begins, not after the first export arrives.

For sources with access controls or anti-bot measures, a responsible provider should assess feasibility rather than promise access. Such measures can indicate that the source does not want automated traffic, and they may create technical, contractual and legal constraints. Read more about the business implications in website anti-bot protection.

What data businesses collect and why

The best web scraping projects answer a recurring decision question. Data without an owner or a decision attached becomes another spreadsheet that nobody opens. Start with the decision, then work backwards to the smallest useful set of fields.

Common public-data use cases

• Price and assortment monitoring: product title, model or SKU, listed price, discount, stock indicator, seller, URL and collection date. Teams use this to review positioning, gaps and availability across public retailer listings. See competitor price monitoring for a typical application.
• Marketplace research: public product listings, categories, seller offers, ratings and review counts. This can inform assortment, catalogue priorities and market-entry research. Relevant sources can be found in the marketplace scraper category.
• Property market analysis: public listing location, property type, asking price, bedroom count, listing status, agent and listing date. Analysts may use the data to track inventory and advertised-price movements, while avoiding personal information.
• Job market intelligence: public job title, employer, location, salary where shown, skills, contract type and posting date. This helps recruitment teams and workforce planners understand visible demand.
• Reputation analysis: public review text, rating, review date and location or product context. The emphasis should be on aggregate patterns, recurring complaints and service themes rather than identifying individual reviewers.
• Directory and local-market research: public company name, category, website, address and publicly listed business contact details, subject to a careful privacy and marketing-law review before any outreach.

Use an identifier wherever possible. A product URL alone may change; a manufacturer part number, SKU or stable listing ID makes matching more reliable. Also preserve the raw source value alongside cleaned fields. If a price appears unusual, an analyst should be able to see the original label and source page rather than rely on a transformed number with no audit trail.

Choosing CSV, Excel, JSON, Google Sheets, an API or webhook

Output format affects who can use the data and how quickly it becomes useful. The right choice is often driven by the receiving team’s workflow rather than the scraper itself.

• CSV works well for large, simple tables and imports into analytics tools, databases and many business systems. It is compact but does not preserve spreadsheet formulas or rich formatting.
• Excel is useful when commercial teams need to open, filter, annotate and share a familiar file. It is usually best for a finite export or human review.
• JSON suits nested records and developer-led integrations. A property record with photos, features and multiple agents, for example, may fit JSON better than a flat table.
• Google Sheets is practical for shared operational tracking, smaller datasets and teams that already use spreadsheet-based processes. Define who owns edits to avoid overwriting source data.
• An API is appropriate when another application needs to request structured data programmatically. It requires a clear schema, authentication approach and expectations for updates.
• A webhook pushes each completed batch or event to a destination automatically. It can reduce manual handling when a CRM, database or internal workflow needs fresh records.

For recurring projects, request a data dictionary: a short document that defines every field, data type, allowed values, update timing and how missing values are represented. This avoids problems such as interpreting an empty stock field as “in stock” or treating a missing price as zero. Kodrana can deliver extraction results as CSV, Excel, JSON, Google Sheets, an API or webhook, or directly into a CRM or database as part of a defined workflow.

Ready scraper, custom scraper or managed service?

There are three practical routes, and the lowest initial cost is not always the lowest operational cost. Consider source stability, volume, update frequency, internal technical capacity and the cost of acting on incorrect or stale data.

When each option fits

• A ready scraper fits when your source and fields are standard, you can operate the tool internally and the output needs little transformation. Kodrana has a catalogue of ready scrapers covering more than 5,400 websites.
• A custom scraper fits when you need a particular site, unusual filters, field logic, record matching or delivery into your own system. It is also useful when the output must reflect your internal data model rather than a generic template.
• A managed data-extraction service fits when your team wants a defined dataset and schedule without maintaining collection logic. It is particularly suitable for recurring monitoring, quality checks and source changes.
• Manual research fits small, infrequent requests where the setup effort would exceed the value of automation. Set a threshold in advance: number of sources, records or repeated reporting cycles at which the work should be automated.

For a one-off export of up to 10,000 records, Kodrana lists prices from $249, $399 or $690 depending on site protection. Weekly updates start from $149/month and daily updates from $299/month, plus one-time setup. The subscription includes fixes within 48 hours when a site changes, data storage and support. For a new site, the stated development time is 2-5 business days; scope, source access and data rules should still be confirmed before committing to a timeline.

A useful procurement brief states what success looks like: the source domains, public pages or search criteria, columns, volume, sample records, schedule, output destination, acceptance checks and named business owner. More detailed budgeting and scoping guidance is available in web scraping project costs.

Limits, legal considerations and operational risks

Publicly visible does not automatically mean unrestricted use. Website terms, intellectual-property rights, privacy law, contract law and technical access rules can all matter. In the United States, the Computer Fraud and Abuse Act can be relevant to unauthorised access questions. In the UK, the Computer Misuse Act 1990 is relevant to unauthorised access. The facts depend heavily on the source, conduct and jurisdiction, so seek advice from a qualified lawyer for your specific use case.

Privacy requires separate attention. If data can identify a living person, GDPR or UK GDPR may apply to EU or UK personal data, and California’s CCPA/CPRA may be relevant for covered businesses and California residents. Even public information can be personal data. Do not assume that a public profile, review or directory entry can be collected and repurposed for prospecting without assessing lawful basis, transparency, retention, security and applicable marketing rules.

Copyright may protect page content and images. In the EU, the database right may also protect substantial investment in obtaining, verifying or presenting database contents. Avoid treating scraped material as content you can republish freely. Collect only what serves the stated analytical purpose, retain it no longer than necessary and keep source attribution where appropriate. For a fuller starting point, see web scraping legal guidance.

• Terms and access: review the website’s terms, robots guidance where available, contractual restrictions and whether an official API is available.
• Technical resilience: page designs, URLs, labels and categories change. Build monitoring, sample checks and a process for repairs into a recurring project.
• Data quality: public listings may be outdated, incomplete, duplicated or inconsistent. Treat web data as an observed signal, not automatically as ground truth.
• Security: restrict access to exports, use appropriate retention periods and avoid mixing external data with sensitive internal datasets without controls.
• Reputation and load: use restrained collection rates, avoid disruption and never try to bypass logins, paywalls or protective controls.

Kodrana states that it collects publicly available data only, does not log into other people’s accounts, does not collect personal data and keeps load on websites gentle. These boundaries are useful project guardrails, but they do not replace a legal review when your intended use, geography or data category creates additional obligations.

How to start a data extraction project

Begin with a small validation run. A sample exposes ambiguous fields, missing values and matching problems before they become expensive at scale. It also lets the commercial owner confirm that the output answers the original question rather than merely looking complete.

1. Write one decision the dataset will support, such as weekly price review or monthly local-market analysis.
2. Name the public sources and record any terms, legal or internal compliance constraints that need review.
3. Define mandatory and optional fields, including source URL, collection time and a stable identifier where available.
4. Choose an initial sample size and acceptance rules, such as required-field completion, duplicate handling and date format.
5. Select a delivery format and recipient: analyst, spreadsheet owner, CRM administrator or engineering team.
6. Decide whether the need is one-off, weekly or daily, and assign someone to review exceptions and data-quality reports.

A scraping project often becomes more valuable when connected to business automation: price alerts, inventory comparisons, lead-routing rules or reporting dashboards. If data should trigger an internal action rather than remain in a file, consider the broader business automation service scope at the planning stage.

A practical next step

If you have a public website source and a defined list of fields, Kodrana offers a free test: send the site and required fields, and the team checks its protection before sending a free sample of 50-100 records before payment. You can then compare the sample with your acceptance criteria and request a quote for a one-off export or recurring delivery, without committing before you see whether the dataset is useful.