How to Scrape 2.8 Million Company Records from Crunchbase Using Sitemaps and Google Cache

Crunchbase holds one of the largest public databases of company information on the internet -- funding rounds, employee counts, locations, websites, and more across millions of organizations. The catch is that accessing it at scale normally means paying for their API or enterprise plans. In this walkthrough, @thewebscrapingguy demonstrates how he extracted 2.8 million company records using a three-step approach that leans on Crunchbase's own sitemap files and Google's cached pages.

Step 1: Find Every Organization URL via Sitemaps

The starting point is Crunchbase's sitemap. A Google search for "Crunchbase sitemap" leads to their sitemap index at crunchbase.com/www-sitemaps/sitemap-index.xml, which contains dozens of compressed XML files named sitemap-organizations-1.xml.gz through sitemap-organizations-32.xml.gz (and beyond). Each XML file contains thousands of organization URLs in a standard sitemap format, with entries like https://www.crunchbase.com/organization/greater-austin-allergy alongside metadata like last-modified dates and change frequency.

Downloading and parsing all of these sitemap files gives you a complete list of every public organization page on Crunchbase -- roughly 2.8 million URLs.

Step 2: Bypass Rate Limiting with Google's Cached Pages

Hitting Crunchbase directly at scale will get you blocked quickly. The workaround is to fetch Google's cached version of each page instead. Google regularly crawls and stores snapshots of public web pages, and those cached copies are accessible through a specific URL pattern. By routing requests through Google's cache rather than Crunchbase's servers directly, you avoid Crunchbase's anti-scraping protections entirely.

Step 3: Extract Structured Data with Cheerio

Each Crunchbase organization page embeds structured JSON data in an ng-state element in the HTML. Rather than trying to parse the rendered page layout, the scraper uses Cheerio (a server-side HTML parser for Node.js) to grab that single element. The embedded JSON contains the full company profile -- name, description, location, employee count, funding stage, website, and more. The video briefly shows this in action on Shopify's Crunchbase profile, which includes fields like headquarters (Ottawa, Ontario, Canada), employee count (10,001+), funding stage (Series C), public/private status, and website URL.

The JavaScript code to orchestrate this is straightforward: loop through every URL from the sitemap files, fetch the cached version of each page, parse out the ng-state JSON with Cheerio, and write the results to a database. The end result is 2.7 million+ records stored locally.

Key Takeaways

• Sitemaps are an underused entry point for large-scale scraping. Any site that publishes XML sitemaps is handing you a complete index of their content. Crunchbase's sitemap alone contains 32+ organization files covering millions of URLs.
• Google's page cache is a practical workaround for anti-bot protections. Instead of fighting rate limits and CAPTCHAs on the target site, fetching the cached version sidesteps those defenses.
• Embedded JSON beats HTML parsing. Many modern web apps (especially Angular-based ones like Crunchbase) embed their data as JSON in the page source. Targeting that structured data directly is more reliable and simpler than scraping rendered HTML.
• The toolchain is minimal. This entire pipeline runs on JavaScript with Cheerio for HTML parsing. No headless browsers, no Selenium, no complex infrastructure -- just HTTP requests and XML/HTML parsing.