Firecrawl website crawl
Crawl multiple URLs based on options. Needs url.
Inputs
- URL
- The base URL to start crawling fromRequired · url
- Prompt
- A prompt to use to generate the crawler options (all the parameters below) from natural languageOptional · prompt
- Exclude paths
- URL pathname regex patterns that exclude matching URLs from the crawlOptional · excludePaths
- Include paths
- URL pathname regex patterns that include matching URLs in the crawlOptional · includePaths
- Max discovery depth
- Maximum depth to crawl based on discovery orderOptional · maxDiscoveryDepth
- Sitemap
- Sitemap mode when crawling (skip, include or only)Optional · sitemap
- Ignore query parameters
- Do not re-scrape the same path with different (or none) query parametersOptional · ignoreQueryParameters
- Regex on full URL
- When true, includePaths and excludePaths regex patterns are matched against the full URL (including query parameters)…Optional · regexOnFullURL
- Limit
- Maximum number of pages to crawlOptional · limit
- Crawl entire domain
- Allows the crawler to follow internal links to sibling or parent URLs, not just child pathsOptional · crawlEntireDomain
- Allow external links
- Allows the crawler to follow links to external websitesOptional · allowExternalLinks
- Allow subdomains
- Allows the crawler to follow links to subdomains of the main domainOptional · allowSubdomains
- Ignore robots txt
- Ignore the website's robots.txt rulesOptional · ignoreRobotsTxt
- Robots user agent
- Custom User-Agent string for robots.txt evaluationOptional · robotsUserAgent
- Delay
- Delay in seconds between scrapesOptional · delay
- Max concurrency
- Maximum number of concurrent scrapesOptional · maxConcurrency
- Webhook
- A webhook specification objectOptional · webhook
- Scrape options
- An objectOptional · scrapeOptions
- Zero data retention
- If true, this will enable zero data retention for this crawlOptional · zeroDataRetention
Pricing
- Price
- Usage based, at the cost the provider reports for the call
- Billing
- Billed at the cost the provider reports for the call, never above $625. Your agent sees the maximum before running.
- Price checked
- 5 October 2026
Call
Your agent reads the schema and the price with inspect_tool before it runs the tool.
- MCP call
- cURL command
- TypeScript
- Tool ID
Alternatives
- Create a Firecrawl monitorFirecrawlUsage based
- Delete a Firecrawl monitorFirecrawlFree
- Firecrawl crawl errorsFirecrawlFree
- Firecrawl monitor check detailsFirecrawlFree
- Firecrawl monitor checksFirecrawlFree
- Firecrawl monitor detailsFirecrawlFree
- Firecrawl website mapFirecrawlfrom $0.0025
- Run a Firecrawl monitorFirecrawlUsage based
- Compare all 10
Connect your agent to 2,400+ tools for free
Sign in with Google or GitHub. No card required.