Crawlee Python

A web scraping and browser automation library

Crawlee covers your crawling and scraping end-to-end and helps you build reliable scrapers. Fast.

Your crawlers will appear almost human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.

👉 View full documentation, guides and examples on the [Crawlee project website](https://crawlee.dev/python/) 👈

We also have a TypeScript implementation of the Crawlee, which you can explore and utilize for your projects. Visit our GitHub repository for more information Crawlee for JS/TS on GitHub.

Installation

We recommend visiting the Introduction tutorial in Crawlee documentation for more information.

Crawlee is available as `crawlee` package on PyPI. This package includes the core functionality, while additional features are available as optional extras to keep dependencies and package size minimal.

To install Crawlee with all features, run the following command:

python -m pip install 'crawlee[all]'

Then, install the Playwright dependencies:

playwright install

Verify that Crawlee is successfully installed:

python -c 'import crawlee; print(crawlee.__version__)'

For detailed installation instructions see the Setting up documentation page.

With Crawlee CLI

The quickest way to get started with Crawlee is by using the Crawlee CLI and selecting one of the prepared templates. First, ensure you have uv installed:

uv --help

If uv is not installed, follow the official installation guide.

Then, run the CLI and choose from the available templates:

uvx 'crawlee[cli]' create my-crawler

If you already have crawlee installed, you can spin it up by running:

crawlee create my-crawler

Examples

Here are some practical examples to help you get started with different types of crawlers in Crawlee. Each example demonstrates how to set up and run a crawler for specific use cases, whether you need to handle simple HTML pages or interact with JavaScript-heavy sites. A crawler run will create a storage/ directory in your current working directory.

BeautifulSoupCrawler