How to Monitor Competitor Prices: Comparing Methods and Building a System

Three Ways to Collect Competitor Prices

Businesses use three main approaches for competitor price monitoring, each with different trade-offs in cost, flexibility, and maintenance.

Manual Price Checks

Someone visits competitor websites, copies prices into a spreadsheet, and updates it weekly or monthly. This works for small catalogs—under fifty SKUs—and occasional spot checks. The advantage is zero upfront cost and no technical setup. The downsides are speed (an hour to check twenty products across five sites), human error, and the inability to track historical trends or catch flash sales. Manual checks make sense for high-value items with stable pricing, like industrial equipment or bespoke services, but break down quickly for retail, where hundreds of products change daily.

Price Monitoring Software

SaaS platforms scrape competitor sites and present the data in dashboards. Most offer product matching, alerts, and integrations with repricing tools. These tools work well when your competitors are on popular platforms like Amazon, eBay, or major UK supermarkets—sites the software already supports. Limitations appear when you need to track niche sites, regional competitors, or specific data points the platform does not collect, such as shipping costs by region or bundle discounts. You also depend on the vendor's update schedule and cannot customize matching rules beyond what their interface allows.

Custom Web Scrapers

A custom scraper is code built to extract data from specific competitor sites and deliver it in the exact format your team needs. It can monitor any public site, collect any visible field, run on your schedule, and feed directly into your database or BI tool. Setup costs more than a SaaS subscription—typically from $399 for a single-site scraper plus ongoing updates—but you own the process. This approach fits businesses with unique competitor sets, complex product catalogs, or specific compliance requirements. The trade-off is maintenance: when a site redesigns, the scraper needs updating, though many providers include fixes in their subscription.

Most mid-sized retailers combine methods: software for mainstream competitors, custom scrapers for regional or niche players, and occasional manual checks for outliers.

Product Matching: Linking Your SKU to Theirs

Collecting prices is only useful if you know which of your products corresponds to which competitor listing. Matching is the hardest part of competitor price tracking and where most manual efforts fail.

Standard Identifiers: EAN, UPC, GTIN

European Article Numbers (EAN), Universal Product Codes (UPC), and Global Trade Item Numbers (GTIN) are numeric barcodes that uniquely identify products. If your catalog and your competitors both use them, matching is automatic and reliable. This works well for consumer electronics, books, branded groceries, and other manufactured goods sold through multiple retailers. The catch is that not all sites display these codes publicly—some bury them in page metadata or omit them entirely—and private-label or unbranded items often lack standard identifiers altogether.

Manufacturer Part Numbers and SKUs

Manufacturer part numbers (MPNs) are another reliable identifier, especially in automotive, industrial, and B2B markets. Unlike EAN codes, MPNs are not always globally standardized, so the same part might appear with variations. Internal SKUs are useless for matching unless you maintain a mapping table that links your SKU to each competitor's SKU, which requires manual setup and ongoing maintenance as competitors add or discontinue products.

Attribute-Based Matching

When identifiers are missing or unreliable, you match on product attributes: brand, model, size, color, and other specifications. A scraper extracts these fields, and your system compares them to your catalog using exact or fuzzy matching. For example, 'Samsung Galaxy S23 128GB Black' needs to match 'Samsung S23 (128GB) - Black' even though the formatting differs. Attribute matching introduces ambiguity—two products might be similar but not identical—so most systems assign a confidence score and flag uncertain matches for manual review. This method is common in fashion, furniture, and any category where products lack barcodes.

In practice, you use a hierarchy: match on EAN or MPN first, fall back to attributes, and manually verify matches for high-value items or new product launches.

Update Frequency and What Drives It

How often you need fresh data depends on your market's volatility and your repricing strategy. More frequent updates cost more in computation and risk triggering anti-bot measures, so the goal is finding the minimum frequency that keeps you competitive.

• Hourly or real-time: Used in highly competitive categories like consumer electronics during sale events, or dynamic markets like travel and fuel. Custom scrapers can run every few hours if site protections allow.
• Daily: The standard for e-commerce and retail. Prices change overnight, and daily updates let your team adjust before the business day starts. This is the sweet spot for cost and responsiveness.
• Weekly: Sufficient for stable markets like B2B supplies, furniture, or automotive parts, where prices move slowly and relationships matter more than a few percentage points.
• Monthly or on-demand: Works for spot checks, audits, or industries with contract-based pricing that changes infrequently.

Your update schedule also affects how you handle site protections. Aggressive scraping—hitting the same site every hour—raises the likelihood of rate limiting or CAPTCHAs. Rotating user agents, spacing requests, and using proxies help, but staying within a site's implicit tolerance is part of responsible data collection. If you are collecting from dozens of competitors, stagger the schedule so not all scrapers run simultaneously.

Tracking Stock Levels and Promotions

A competitor's price is only meaningful if the product is actually available. Stock status and promotional flags add context that prevents bad decisions.

Most e-commerce sites display stock levels in some form: 'In Stock', 'Low Stock', 'Out of Stock', 'Pre-Order', or specific quantities. Your scraper should capture this field along with the price. If a competitor shows £199 but the item is out of stock, they are not really competing on that SKU right now, and you should not drop your price to match. Some retailers hide stock status from scrapers or show generic 'Available' messages; in those cases, you infer availability from whether the 'Add to Cart' button is active or the product redirects to a discontinuation page.

Promotions are harder to standardize. Discounts might appear as strikethrough prices, percentage badges, coupon codes, or bulk-buy offers. A good scraping setup extracts both the current price and the original price (if shown), any promotional text, and the validity period if visible. This lets your analysts distinguish between a permanent price drop and a weekend flash sale. Some retailers use JavaScript to calculate discounts client-side, so your scraper needs to render the page or execute scripts to see the final number.

In the UK grocery sector, multi-buy offers like 'Buy 2 for £5' or 'Meal Deal' bundles complicate comparisons. Unless your system extracts and normalizes these to a per-unit price, you will miss that a competitor's higher list price is actually cheaper when the promotion applies.

Legal and Ethical Boundaries in Retail Price Monitoring

Collecting publicly visible prices from competitor websites is generally lawful in the US and UK, but you must respect access rules and data protection laws.

In the US, the Computer Fraud and Abuse Act (CFAA) prohibits accessing a computer system without authorization. Courts have interpreted this narrowly: reading public web pages is authorized, but bypassing technical barriers, ignoring robots.txt, or using stolen credentials is not. Aggressive scraping that causes service disruption can also create liability under tort law. California's CCPA and other state privacy laws restrict the collection of personal data; in a pricing context, this is rarely an issue unless reviews or seller contact details are involved.

In the UK and EU, the Computer Misuse Act and GDPR set similar boundaries. Scraping product catalogs and prices does not typically involve personal data, but if your scraper collects seller names, email addresses, or user reviews, GDPR's lawful basis and data minimization principles apply. The EU Database Directive protects databases as intellectual property, though extracting insubstantial parts for analysis is generally considered fair use. If you scrape a large portion of a competitor's catalog repeatedly, legal risk increases.

Practically, follow these rules: respect robots.txt and rate limits, do not create server load that could be seen as a denial-of-service, never log into accounts that are not yours, and avoid collecting personal data unless you have a lawful basis and inform data subjects. Most competitor monitoring falls comfortably within legal boundaries when done responsibly, but if your use case involves scraping behind logins, accessing price data marked as confidential, or integrating personal customer data, consult a lawyer before proceeding.

Delivering Data to Your Team: Formats and Integration

Raw price data is useless until it reaches the people who make repricing decisions. Delivery format and integration determine whether your monitoring system gets used or ignored.

Spreadsheets and Dashboards

For small teams, a daily CSV export or a Google Sheets integration is often enough. The scraper runs overnight, appends new prices to a sheet, and the pricing manager reviews it over morning coffee. This works well for manual repricing or categories where prices change slowly. The downside is that spreadsheets do not scale—once you are tracking thousands of SKUs across dozens of competitors, filtering and pivoting becomes slow and error-prone.

BI Tools and Databases

Mid-sized retailers push scraped data into business intelligence platforms like Tableau, Power BI, or Looker, or directly into a SQL database that feeds their internal dashboards. This lets analysts slice by category, compare trends over time, and set alerts when a competitor undercuts you by more than a threshold percentage. A custom scraper can deliver JSON or CSV to an API endpoint, write directly to a PostgreSQL or MySQL database, or trigger a webhook that your system consumes. This approach requires more setup but scales to hundreds of thousands of products.

Repricing Automation

Some businesses feed competitor prices directly into automated repricing engines. The scraper updates a database, the repricing software reads it, applies rules like 'stay 2% below the lowest competitor' or 'match the market leader but never go below cost', and pushes new prices to your e-commerce platform via API. This is common on Amazon, eBay, and other marketplaces where prices change multiple times per day. The risk is that without safeguards—minimum margins, maximum price swings, manual approval for large SKUs—you can end up in a race to the bottom or accidentally price a product at a loss due to a scraper error.

Whichever format you choose, include metadata in the export: timestamp, competitor name, product match confidence, stock status, and any warnings. This context helps your team trust the data and investigate anomalies.

Building a Custom System and Working with Kodrana

If off-the-shelf tools do not cover your competitors or your integration needs are specific, a custom system gives you full control. The typical build starts with listing your competitors and the data points you need—price, stock, promotions, product identifiers, shipping costs. Prioritize by impact; you do not need to track every field from day one. Next, map your product catalog to competitor SKUs, identifying which products have EAN or MPN codes and which need attribute matching. This step takes longer than the scraping itself. Choose a delivery method—database, API, spreadsheet, or BI tool—because this decision affects how you structure the scraper output and where you store historical data.

Build or commission scrapers for each competitor site. If you are using ready-made scrapers, check whether your competitors are already covered. If not, a developer can typically build a scraper for a new site in two to five business days. Set up scheduling and monitoring: decide update frequency, configure retries for transient failures, and add alerts for when a scraper stops returning data or a competitor site blocks your requests. Validate and iterate by running the system for a week in parallel with your existing process. Check match accuracy, spot-check prices manually, and adjust matching rules and scraping logic based on what you find.

Ongoing maintenance is part of the cost. Competitor sites redesign every few months, breaking scrapers. A reliable monitoring system includes automated checks that detect when scraped data looks wrong—sudden price drops to zero, missing fields, or no updates for several days—and alerts you to investigate.

Kodrana builds competitor price monitoring systems from $399 plus a subscription for daily or weekly updates. Before you pay, send us your competitor sites and the fields you need; we check the site protection and send a free sample of 50–100 records. Delivery is as CSV, Excel, JSON, Google Sheets, API, or directly into your database or BI tool. Subscription includes fixes within 48 hours when sites change.