Crawlee: A Web Scraping Library That Handles the Hard Parts

Crawlee is an open-source web scraping and browser automation library with over 16,000 developers using it. It handles proxy rotation, browser management, and JavaScript-heavy sites out of the box, so you can focus on the data extraction logic instead of the infrastructure.

What Crawlee Does

The library covers the full scraping pipeline. It integrates with Playwright and Puppeteer for rendering JavaScript-heavy pages, manages browser instances automatically, and rotates proxies to keep scrapers from getting blocked.

The documentation walks through everything from basic crawlers that print to the console up to full-featured scrapers that collect links from sites like Amazon.

Package Structure and Setup

Crawlee ships as an NPM package and requires Node.js 16 or higher. The project recently consolidated its package structure, grouping all crawler classes into a single crawlee.crawlers sub-package for cleaner imports.

Getting started takes a single terminal command. The built-in CLI scaffolds a new project with the configuration and boilerplate already in place.

Key Takeaways

• Crawlee handles proxy rotation and browser management automatically
• Works with Playwright and Puppeteer for JavaScript-heavy sites
• Single CLI command to scaffold a new scraping project
• Available as an NPM package with a Python version in early access

Resources

• Crawlee GitHub -- Source code and README
• Crawlee Documentation -- Full guides, API reference, and examples