How to scrape Amazon, eBay and other marketplaces: data collection for sellers and brands

Which marketplaces matter in the US and UK

Amazon dominates both markets, accounting for roughly 38 percent of US e-commerce sales and a similar share in the UK. eBay remains the second-largest platform by active listings, particularly strong in electronics, collectibles, and refurbished goods. Walmart Marketplace has grown rapidly in the US, while Etsy captures handmade and vintage niches on both sides of the Atlantic.

UK-specific platforms include Argos for general merchandise, Currys for electronics, and grocery comparison sites like Trolley. In the US, regional players such as Newegg (tech), Wayfair (furniture), and Chewy (pet supplies) hold category strength. Food delivery marketplaces—Deliveroo and Just Eat in the UK, DoorDash and Uber Eats in the US—also generate competitive intelligence needs for restaurant chains tracking menu pricing and placement.

Brands selling across multiple channels need unified data: a price drop on Amazon should trigger repricing on eBay and your own Shopify store within hours, not days. Retailers monitoring competitors face the same multi-platform reality.

What data sellers and brands collect from marketplaces

Pricing remains the most common target. An automated price monitoring workflow tracks your SKU and competitor ASINs every few hours, flags undercuts, and feeds a repricing engine. You capture list price, sale price, shipping cost, Prime eligibility, and any volume discounts or coupon codes displayed on the page.

Stock and availability signals matter when you're deciding whether to raise your own price or when scouting supply-chain disruptions. Out-of-stock competitors create short windows to capture market share; knowing which sellers still have inventory lets you adjust bids or ad spend in real time.

Seller counts and Buy Box ownership tell you how crowded a listing is. If fifteen sellers compete on the same ASIN and the Buy Box rotates hourly, aggressive pricing becomes mandatory. Conversely, exclusive or limited-competition listings support higher margins.

Review metrics—overall star rating, review count, recent review velocity, and keyword frequency in review text—guide product development and marketing copy. A spike in one-star reviews mentioning 'flimsy hinge' on a competitor's laptop case signals a feature you can emphasize in your own listing.

Search rankings and sponsored-ad positions reveal which keywords competitors bid on and how much traffic those terms drive. Scraping the first three pages of search results for 'wireless earbuds' twice daily shows you whose organic rank is climbing and who just increased ad spend.

Product catalogues and attribute data help marketplace aggregators, comparison engines, and private-label brands. You pull title, bullet points, description, technical specifications, image URLs, brand name, and category breadcrumbs to build a clean database or train a recommendation model.

Typical tasks and use cases for marketplace scraping

Dynamic repricing and MAP enforcement

Brands with minimum advertised price policies scrape reseller storefronts daily to catch violations. When a reseller lists below MAP, the brand's compliance team sends a warning or suspends supply. Authorized sellers use the same data to stay competitive without breaching agreements.

Assortment gaps and white-label opportunities

Private-label sellers analyze category bestsellers, filter by review count and rating, then cross-reference supplier catalogs on Alibaba to find manufacturable variants. A marketplace scraper that captures sales rank, price history, and review sentiment shortens product-selection cycles from weeks to days.

Ad intelligence and keyword research

Tracking which ASINs appear in sponsored slots for high-volume keywords shows you where competitors allocate budgets. Combine that data with your own conversion rates to model whether bidding on the same terms will be profitable.

Inventory arbitrage and dropshipping

Resellers compare prices across Amazon, eBay, and Walmart in near-real-time, buying low on one platform and listing higher on another. Speed matters: arbitrage windows close within hours when other bots notice the same spread.

Market-size estimation and trend analysis

Investors and analysts scrape entire categories—every listing under 'camping tents' or 'protein powder'—to estimate total market size, average selling price, and seasonal demand curves. The dataset feeds financial models or guides product-line expansion.

Official seller APIs versus scraping Amazon and eBay prices

Amazon's Selling Partner API (SP-API) gives enrolled sellers access to their own inventory, orders, and fees. The Product Advertising API serves affiliate marketers with basic product details and buy-box prices, but it doesn't reveal competitor pricing, stock levels at other sellers, or review breakdowns. Rate limits sit around one request per second for most endpoints, and you need an approved developer account tied to a seller or affiliate ID.

eBay's REST APIs offer similar constraints: sellers retrieve their own listings and order data, while the Browse API gives public product searches at roughly five calls per second for standard-tier applications. Neither API exposes the full set of active sellers on a multi-seller listing, historical price changes, or completed-auction hammer prices beyond the past ninety days.

Walmart's Marketplace API focuses on seller onboarding, inventory feeds, and order management. Pricing data for competitors remains unavailable. Etsy's Open API provides shop and listing metadata but omits sales volume and most search-ranking signals.

When an official API meets your needs—syncing your own inventory to a dashboard, for example—use it. The data arrives structured, the rate limits are documented, and you avoid terms-of-service ambiguity. When you need competitor intelligence, search-result positions, or cross-seller comparisons, scraping becomes the only practical option.

Anti-bot protection and rate limits on major marketplaces

Amazon rotates between multiple protection layers: CAPTCHA challenges appear after a few dozen rapid requests from the same IP, and persistent scrapers encounter HTTP 503 errors or temporary IP bans lasting minutes to hours. The site serves different HTML structures to different user-agents, and some pages load product details via JavaScript fetch calls rather than embedding them in the initial response.

eBay uses similar defenses—rate-based throttling, CAPTCHA gates, and fingerprinting that correlates request headers, TLS signatures, and mouse-movement patterns. Scraping search results at more than one page per few seconds triggers a temporary block.

Walmart Marketplace employs Akamai bot-management scripts that inspect browser automation markers. Requests from data-center IPs face stricter limits than residential proxies. Etsy has lighter protection but still blocks obvious bot traffic after sustained high-frequency polling.

Effective scraping respects these controls: rotate residential proxies or use a proxy pool, randomize request intervals, render JavaScript where required, and mimic realistic session behavior—adding items to cart, browsing related products—rather than hammering product pages in sequence. A well-designed Amazon scraper stays under the radar by spreading requests across time and IP space, keeping the load gentle and the data flow consistent.

Legal and compliance considerations in the US and UK

US courts have generally permitted scraping of publicly accessible data under the Computer Fraud and Abuse Act, provided you do not bypass technical access controls such as login walls or paywalls. The Ninth Circuit's hiQ Labs v. LinkedIn decision affirmed that scraping public profiles does not violate the CFAA, though the case addressed LinkedIn's attempt to block a specific scraper rather than marketplace data collection.

California's CCPA and its successor CPRA restrict the sale and sharing of personal information, but product listings, prices, and aggregated review counts rarely qualify as personal data. If you do collect seller contact details or customer reviews that include names, consult a privacy lawyer before using or sharing that data.

In the UK, the Computer Misuse Act 1990 criminalizes unauthorized access to computer systems. Scraping public product pages typically does not meet the threshold for 'unauthorized access,' but bypassing a login or ignoring an explicit IP block might. UK GDPR applies the same personal-data protections as the EU regulation: seller business names and publicly listed contact details fall outside the scope, while customer email addresses harvested from reviews do not.

The EU Database Directive grants database makers sui generis rights over substantial compilations. A UK or EU-based marketplace could argue that scraping its entire catalog infringes that right, though enforcement remains uneven and fact-specific. US law offers no equivalent database right, but terms of service often prohibit automated data collection, creating a contract-law risk rather than a statutory one.

Practical guidance: collect only data that appears on public pages without login, avoid personal information unless you have a lawful basis under GDPR or CCPA, keep request rates reasonable, and review the legal framework with a lawyer if your use case involves high volume or sensitive categories.

Technical trade-offs: building versus buying a marketplace scraper

A small in-house script works when you track a few dozen ASINs, run checks once daily, and can tolerate occasional downtime when Amazon changes its HTML. Most teams underestimate maintenance: marketplaces redesign pages every few months, shift data into JavaScript-rendered components, or roll out new bot-detection rules. Each break requires developer time to diagnose, patch, and redeploy.

Cloud scraping platforms charge per request or per record and handle infrastructure, proxies, and CAPTCHA solving. Monthly costs range from a few hundred dollars for light use to several thousand for high-frequency monitoring. You gain speed and reliability but sacrifice control over request logic and data-cleaning pipelines.

A custom scraper built by a specialized studio combines control and maintenance. Kodrana's catalog includes ready solutions for Amazon, eBay, and other marketplaces; a new scraper for a platform not yet covered takes two to five business days. Pricing starts at £149 for a one-off export of up to ten thousand records, or from £99 per month for weekly updates with fixes included when the site changes. You receive data in CSV, JSON, or directly into your CRM, and the studio handles proxy rotation, rate limiting, and parser updates.

The decision hinges on volume, frequency, and internal capacity. If your team can dedicate ongoing developer hours, an in-house build makes sense for highly custom workflows. If you need reliable data without engineering overhead, a scraping service delivers faster and often cheaper over twelve months.

Setting up a marketplace data workflow

Start by defining exactly which data points drive decisions. A repricing tool needs current list price, shipping cost, and Buy Box status every few hours. A product-research dashboard needs review counts, star ratings, and search rank once daily. Overcollecting fields increases cost and noise without improving outcomes.

1. Identify target ASINs, product IDs, or search keywords. Export your current catalog or competitor list as a CSV with unique identifiers.
2. Choose update frequency. Hourly updates suit high-velocity categories like electronics; daily checks suffice for slower-moving goods like furniture.
3. Select delivery format. Direct API or webhook integration feeds your repricing engine in real time; a nightly CSV drop into Google Sheets works for manual analysis.
4. Test with a small batch. Scrape fifty products for a week to confirm data accuracy and spot any missing fields before scaling to thousands of SKUs.
5. Monitor for errors. Set alerts when the scraper returns empty fields, duplicate records, or HTTP errors, and have a process to fix breaks within forty-eight hours.

A price-monitoring project with Kodrana begins with a free sample: you send the marketplace URLs and required fields, the team checks protection and delivers fifty to one hundred records before any payment. That sample confirms data quality and structure, and the fixed-price quote covers setup, subscription, and guaranteed fixes when Amazon or eBay change their layouts.

Getting started with marketplace data collection

If you're tracking competitors on Amazon, eBay, or another platform and manual checks have outgrown spreadsheets, an automated scraper turns hours of copy-paste work into a scheduled data feed. Kodrana offers a free test sample—send the product URLs and fields you need, and you'll receive fifty to one hundred records to verify quality before committing. Subscriptions start at £99 per month with weekly updates and include parser maintenance when sites change, so your data pipeline stays live without surprise engineering costs.