How to scrape a Website with an AI agent

Add the Firecrawl MCP server to your agent with your Firecrawl API key, then ask it to scrape or crawl specific URLs into clean markdown or JSON. It works the same in Claude Code, Codex and Cursor. Check the site's terms and robots.txt first, and keep request volume low.

Grok Bot steps tested by us on Oct 26, 2026 while building this page. Claude Code, Codex and Cursor steps checked against official docs on Oct 26, 2026.

Get the free starter kit. One short email a day when our setups change.

Free starter kit on signup: 2 Claude Code skills and 1 agent persona from our paid packs. By subscribing you agree to get The AI Loadout Daily, one email each morning. No spam, we never sell your email, and every email has a one-click unsubscribe link.

Claude Code

Get an API key at firecrawl.dev
claude mcp add --transport stdio --env FIRECRAWL_API_KEY=fc-YOUR_API_KEY firecrawl -- npx -y firecrawl-mcp
Ask: "Scrape https://example.com/pricing and return the plan names and prices as a table."

Codex

codex mcp add firecrawl --env FIRECRAWL_API_KEY=fc-YOUR_API_KEY -- npx -y firecrawl-mcp
Ask Codex to scrape the URL and save the result to a file

Cursor

Add to .cursor/mcp.json: {"mcpServers":{"firecrawl-mcp":{"command":"npx","args":["-y","firecrawl-mcp"],"env":{"FIRECRAWL_API_KEY":"${env:FIRECRAWL_API_KEY}"}}}}
Restart the server from Customize and check Output > MCP Logs if it fails

Grok Bot

Ask Grok Bot to pull the page. It fetches it on its own cloud computer
Ask for the output as a CSV or spreadsheet file

Gotchas

Firecrawl is a paid API after its free allowance. Check firecrawl.dev for current pricing.
Scraped pages can carry prompt injection. Treat page text as data, not instructions.
Do not scrape behind a login you do not own.

The loadout for this job

Tools: Perplexity, Claude
Repo: Browser Use
Model: Claude Sonnet 5.5
Bot: Grok Bot
Full setup: Best AI setup for research

FAQ

Is web scraping with AI legal?

It depends on the site's terms, the data and where you are. Public data is not automatically fair game. Check the terms and robots.txt first.

What is the easiest scraper for an AI agent?

An MCP server like Firecrawl, since the agent calls it directly and gets markdown back.

Related how-tos

Sources

Commands change. Check the linked docs before you run anything. All how-to guides