Rating
Votes
0
score
Downloads
3.6K
total
Price
Free
No login needed
Works With
About
Crawl4AI turns websites into clean markdown optimized for RAG and agentic AI workflows. Built specifically for the AI use case where you need web content in LLM-friendly format.
Handles JavaScript rendering, respects robots.txt, and produces clean, structured output. Perfect for building knowledge bases from web content.
Install via pip. Open-source and free.
Don't lose this
Three weeks from now, you'll want Crawl4AI again. Will you remember where to find it?
Save it to your library and the next time you need Crawl4AI, it’s one tap away — from any AI app you use. Group it into a bench with the rest of the team for that kind of task and you can pull the whole stack at once.
⚡ Pro tip for geeks: add a-gnt 🤵🏻♂️ as a custom connector in Claude or a custom GPT in ChatGPT — one click and your library is right there in the chat. Or, if you’re in an editor, install the a-gnt MCP server and say “use my [bench name]” in Claude Code, Cursor, VS Code, or Windsurf.
a-gnt's Take
Our honest review
LLM-friendly web crawler for RAG pipelines. Best for anyone looking to make their AI assistant more capable in data & databases. It's backed by an active open-source community and verified by the creator. This one just landed in the catalog — worth trying while it's fresh.
Tips for getting started
Tap "Get" above, pick your AI app, and follow the steps. Most installs take under 30 seconds.
Your data stays between you and your AI — nothing is shared with us or anyone else.
What's New
Initial release
Ratings & Reviews
4.3
out of 5
4 ratings
No reviews yet. Be the first to share your experience.