Automated data collection infrastructure for commerce and analytics

Task

We needed a reliable way to collect supplier catalogs and pricing data daily for our own projects. Manual checks were not an option at scale, and different sources had different structures and update schedules.

We also needed to track SEO keywords from analytics services and monitor activity on trading platforms. Each source required its own approach to extraction and storage.

Solution

We built scraping and monitoring systems that run on a schedule. Python and Playwright handle the extraction logic, adapting to different page structures and dynamic content. Data flows into PostgreSQL for storage and querying.

The systems collect supplier catalogs and prices daily, pull SEO keyword data from analytics services, and monitor trading platforms for changes. Each scraper runs in Docker containers, managed by cron for scheduling.

Over time, we assembled a catalog of ready website scrapers. The catalog now includes configurations for different sources, making it straightforward to add new data collection tasks without starting from scratch.

• Daily automated collection of supplier catalogs and pricing
• SEO keyword extraction from analytics services
• Monitoring of trading platforms
• Docker containers for isolated scraper execution
• PostgreSQL storage for structured data access
• Catalog of ready scraper configurations

Result

• Daily collection of supplier catalogs and prices running without manual intervention
• SEO keywords pulled automatically from analytics services
• Trading platforms monitored for activity and changes
• Catalog of ready website scrapers available