HomeFind your caseWe can't get complete data from listing sites
Case · GTM
A common situation · not a client storyWe can't get complete data from listing sites.
Sales, research and data teams that build lists from directories, marketplaces and listing sites, and keep finding gaps in what they collected.
What you're probably seeing
If this sounds familiar, you're in the right place.
- Lists come out shorter than the site says they should be.
- The scraper reports success, but pages behind Load more buttons or endless scroll are missing.
- Runs fail halfway and start again from the beginning.
- Nobody can tell which records are new and which were collected before.
- Each new site needs a new script, written from scratch.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Copy and paste by hand | You need a few hundred records once | Slow, and it doesn't repeat |
| Buy a data provider's list | The provider covers your market well | Coverage and freshness are what they are, and you can't check them |
| Off-the-shelf scraping tools | Sites have simple paging | Stop early on endless scroll, and report success anyway |
| A custom scraper per site | A few sites matter most | Each one needs upkeep when the site changes |
| An official API, where one exists | The site offers the data you need through it | Limits, cost and missing fields |
| A scraper that tracks its own progress | Completeness matters more than speed | Takes longer to build, and must respect each site's terms |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| Office for National Statistics (UK) – web-scraped prices researchNational statistics office. Research project scraping three supermarket websites (about 50% of the UK grocery market) for 33 items daily since June 2014. | Built its own scrapers, then moved toward full-site collection so that changes to a retailer's product menus broke fewer scrapers. Moved collection to AWS, used a browser tool (Selenium) for endless-scroll pages, and filled short gaps by carrying prices forward. | About 7,000 price quotes a day, against about 6,800 a month collected by hand for the same items. Validation removed about a quarter of collected prices on average. Product classification averaged about 85% F1 across the 33 items.They noted: Scrapers stopped collecting an item when a retailer changed its product hierarchy. Using Selenium for endless scroll made full-site runs slow. Some supermarkets could not be added because they block scraping or forbid it in their terms. | Research indices using web scraped price data: August 2017 updateSep 2017 |
| Office for National Statistics (UK) – supplier data feedsHeadline inflation basket of about 750 goods and services, with many prices previously collected by hand. | Switched used cars to Auto Trader listings data and rail fares to Rail Delivery Group data, and planned to use retailer scanner data for about half the grocery market. For the other half of groceries it will keep collecting prices by hand, in store and online. | Used cars: from 105 to 300,000 price points a month. Rail fares: from a single growth rate to around 40 million prices a month. Groceries (planned, for half the market): about 300 million price points a month instead of about 25,000 hand-collected prices.They noted: Full use of grocery scanner data in headline figures was delayed from March 2025 to March 2026, because data of this size and complexity was new to ONS. It chose to run old and new systems in parallel through 2025. | Futureproofing our headline inflation statisticsJan 2025 |
| People Data LabsData vendor selling person, company and job posting datasets. Its job dataset took nearly a year of engineering before the first production release. | Collects job postings straight from company career pages with thousands of custom per-site crawlers run daily, instead of from job boards or aggregators. Each role is timestamped when published, updated and taken down, and closed roles are kept as history. | Not stated (scale given as thousands of crawlers, daily runs, history from October 2024)They noted: They found aggregator data stale, incomplete and full of duplicates and dead posts. Per-site crawling at scale needed dedicated infrastructure and daily quality checks. | How We Source Job Posting DataOct 2025 |
| WoflowCommerce data company syncing merchant menu and inventory data for delivery apps across hundreds of platforms, including UberEats and DoorDash, with thousands of concurrent crawlers. | Moved its crawlers off an outdated Ruby framework onto a hosted scraping platform (Apify), while still writing its own data collection code with Apify's tools. | Adding a new platform went from days to hours. Coverage grew from a handful of platforms to hundreds over a roughly five-year relationship (metrics not dated).They noted: Before the move, each custom crawler took a few days per site and the team spent a lot of time watching infrastructure. The story does not discuss completeness or missed records. | Woflow: Real-time merchant data across 100s of platforms with ApifyOct 2025 · vendor story |
Our read
Same call. Where a data owner offered a feed, ONS took it; where none existed, teams built per-site collection, which is where our tests also point.
ONS's 2017 research hit the issues the diagnostic checks for: scrapers that broke when sites changed, slow handling of endless scroll, and sites ruled out by their terms or blocking. By 2025 it had moved used cars and rail fares to data from the data owners, and it planned to use retailer scanner data for half of groceries from March 2026, which is the official-feed option. People Data Labs and Woflow had no single feed for thousands of career pages or hundreds of platforms, so they built per-site crawlers and accepted the upkeep. None of the write-ups reports how many records each site says it has against how many were collected, which we would still run to confirm completeness.
What we would test first: Compare how many records each site says it has with how many you actually collected.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
How many records each site says it has, against how many you collected.
- 2
How each site loads more results: paging, a button, or endless scroll.
- 3
Where runs fail, and whether they can restart from the last good page.
- 4
Which fields you actually need, and how you'll spot changes between runs.
- 5
Each site's terms and robots rules, before anything runs.
This week, without us
Three things you can do yourself.
- 1
Pick one site and compare its stated result count with what you collected.
- 2
Scroll one listing to the end by hand, and note how it loads more.
- 3
Write down the five fields you really use. Everything else is optional.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.