Crawl4AI

🚀🤖 Crawl4AI: the open-source web crawler for LLMs and AI agents

<div align="center">

<a href="https://trendshift.io/repositories/11716" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

Latest: v0.9.5 (14 Oct 2026) · all releases →

<a href="https://crawl4ai.com/?ref=readme-banner"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-dark.svg"> <img alt="Crawl4AI Cloud is live. Soft launch: free credits every month, no card. Get your key." src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-light.svg" width="960"> </picture> </a> </div>

Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: /scrape a page to Markdown, /extract it to JSON, /search the web, /serp for Google's results page as data, /answer a question with sources; MCP for your agent. Free: 10 a minute, 2 at a time, 1,000 credits every month. Plans: 120 a minute, 10 at a time, from $10 a month.

Two ways to use Crawl4AI

🐍 Run it yourself: open source, forever

pip install -U crawl4ai
crawl4ai-setup        # installs the browser, once

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://news.ycombinator.com")
        print(result.markdown)

asyncio.run(main())

Docker server, CLI and every option: Installation · docs.crawl4ai.com

☁️ Or use the cloud: no browsers, no proxies

Verify your email and 1,000 free credits are yours every month. No card. Soft launch: prices can change, what you buy stays yours. 2. Get any page as Markdown:

curl -s https://api.crawl4ai.com/scrape \
  -H "Authorization: Bearer $CRAWL4AI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown

The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices. 3. Give it to your AI agent. Claude Code shown; Codex, Cursor and OpenCode →

claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
  --header "Authorization: Bearer $CRAWL4AI_KEY"

Which one?

	🐍 Library	🐳 Your own server	☁️ Crawl4AI Cloud
Runs the browsers	you, in your Python process	you, in Docker on your machine	we do
JS-heavy pages and bot walls	your settings, your proxies	your settings, your proxies	handled for you, automatically
Web search	–	–	/search, /serp and /answer
Price	free, forever	free (your hosting)	1,000 free credits every month; plans from $10 a month

<details> <summary>🤓 <strong>My Personal Story</strong></summary>

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

That platform is live now: Crawl4AI Cloud.

</details> <details> <summary>Why developers pick Crawl4AI</summary>

• LLM-ready output: smart Markdown with headings, tables, code and citation hints
• Fast in practice: async browser pool, caching, minimal hops
• Full control: sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence: learns site patterns, explores only what matters
• Deploy anywhere: no keys needed, CLI and Docker, or the hosted cloud

</details>

✨ Features

<details> <summary>📝 <strong>Markdown generation</strong></summary>

• 🧹 Clean Markdown: headings, lists, tables and code blocks, in a structure an LLM reads well.
• 🎯 Fit Markdown: filters remove menus, footers and boilerplate: PruningContentFilterLXML, BM25ContentFilter (for a query) and LLMContentFilter.
• 🔗 Citations: page links become a numbered reference list.
• 🛠️ Your own strategy: plug in a custom Markdown generator.

☁️ Same in the cloud: POST /scrape returns this Markdown, with no browser to run. Docs →

</details> <details> <summary>📊 <strong>Structured data extraction</strong></summary>

• 🔎 CSS and XPath schemas: fast extraction with no LLM (JsonCssExtractionStrategy, JsonXPathExtractionStrategy, RegexExtractionStrategy).
• 🪄 Schema generator: describe what you want once; generate_schema writes a reusable schema.
• 🤖 LLM extraction: any LLM provider, open-source or hosted, into a typed JSON schema (LLMExtractionStrategy).
• 🧱 Chunking: topic, regex and sentence chunking for long pages.
• 🌌 Cosine similarity: find the chunks that match a query (CosineStrategy).

☁️ Same in the cloud: POST /extract, with no LLM key of your own. Docs →

</details> <details> <summary>🌐 <strong>Browser control</strong></summary>

• 🖥️ Your own browser: persistent profiles with saved logins, cookies and settings.
• 🔄 Remote browsers: connect over the Chrome DevTools Protocol (CDP).
• 🔒 Sessions: keep a browser state across multi-step crawls.
• 🧩 Proxies: with authentication and rotation.
• 🕶️ Stealth mode: enable_stealth, and an undetected-browser adapter for sites that detect automation.
• ⚙️ Full control: headers, cookies, user agents, viewport.
• 🌍 Chromium, Firefox and WebKit.

</details> <details> <summary>🔎 <strong>Crawling and scraping</strong></summary>

• 🕸️ Deep crawl: BFS, DFS and best-first strategies, with crash recovery (resume_state) for long crawls.
• 🧠 Adaptive crawling: AdaptiveCrawler stops when it has learned enough to answer your query.
• 🌱 URL discovery: AsyncUrlSeeder (sitemaps, Common Crawl) and DomainMapper; prefetch=True finds URLs 5 to 10 times faster.
• 🚀 Dynamic pages: run JavaScript, wait for elements, scroll the full page (scan_full_page) for infinite scroll and lazy images.
• 📸 Screenshots and PDFs of any page.
• 🖼️ Media and links: images, audio, video, srcset, internal and external links, iframes, metadata.