Skip to content

Crawl and extract a website

Crawl a website and extract page content with path/domain filters. Needs url. Served by Tavily.

Inputs

1 required
URL
The root URL to begin the crawlRequired · url
Instructions
Natural language instructions for the crawlerOptional · instructions
Chunks per source
Chunks are short content snippets (about 800 characters each) pulled directly from the sourceOptional · chunks_per_source
Max depth
Max depth of the crawlOptional · max_depth
Max breadth
Max number of links to follow per level of the tree (i.e., per page)Optional · max_breadth
Limit
Total number of links the crawler will process before stoppingOptional · limit
Select paths
Regex patterns to select only URLs with specific path patterns (e.g., `/docs/.*`, `/api/v1.*`)Optional · select_paths
Select domains
Regex patterns to select crawling to specific domains or subdomains (e.g., `^docs\.example\.com$`)Optional · select_domains
Exclude paths
Regex patterns to exclude URLs with specific path patterns (e.g., `/private/.*`, `/admin/.*`)Optional · exclude_paths
Exclude domains
Regex patterns to exclude specific domains or subdomains from crawling (e.g., `^private\.example\.com$`)Optional · exclude_domains
Allow external
Whether to include external domain links in the final results listOptional · allow_external
Include images
Whether to include images in the crawl resultsOptional · include_images
Extract depth
Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase… (basic or advanced)Optional · extract_depth
Format
The format of the extracted web page content (markdown or text)Optional · format
Include favicon
Whether to include the favicon URL for each resultOptional · include_favicon
Timeout
Maximum time in seconds to wait for the crawl operation before timing outOptional · timeout
Include usage
Credit usage is always requested for actual-cost billingOptional · include_usage

Pricing

Price
Usage based, at the cost the provider reports for the call
Billing
Billed at the cost the provider reports for the call, never above $0.48. Your agent sees the maximum before running.
Price checked
3 November 2026

Call

Your agent reads the schema and the price with inspect_tool before it runs the tool.

MCP call
cURL command
TypeScript
Tool ID

Alternatives

Website crawl & map

Connect your agent to 3,000+ tools for free

Sign in with Google or GitHub. No card required.