# Crawl and extract a website

Crawl a website and extract page content with path/domain filters. Needs url. Served by Tavily.

Crawl a website and extract page content with path/domain filters. The provider receives limit (1 to 100 processed URLs). Cost combines mapping (1 credit per 10 pages, 2 with instructions) and extraction (1 credit per 5 successful URLs, 2 in advanced depth). Billed from usage.credits, including zero. Missing billing counters require reconciliation.

- Tool ID: `tavily/crawl`
- Provider: [Tavily](/providers/tavily.md)
- Capability: [Website crawl & map](/tools/web-scraping/crawl.md)
- Price: usage based, the maximum is shown before a run
- Authentication: an authorised Agentik workspace is required for execution, including zero-price tools.

## Pricing contract

Billed at the cost the provider reports for the call, never above $0.48. Your agent sees the maximum before running.


```json
{
  "model": "reported",
  "costPath": "",
  "costTerms": [
    {
      "path": "usage.credits",
      "usd": 0.008
    }
  ],
  "requireCost": true,
  "maxUsdDefault": 0.016,
  "maxUsdCap": 0.48,
  "headroom": 1,
  "multipliers": [
    {
      "addUsdPer": {
        "field": "limit",
        "usd": 0.008,
        "minus": 10,
        "per": 10,
        "default": 10
      }
    },
    {
      "addUsdPer": {
        "field": "limit",
        "usd": 0.008,
        "minus": 5,
        "per": 5,
        "default": 10
      }
    },
    {
      "when": {
        "field": "instructions",
        "minLength": 1
      },
      "addUsd": 0.008
    },
    {
      "when": {
        "field": "instructions",
        "minLength": 1
      },
      "addUsdPer": {
        "field": "limit",
        "usd": 0.008,
        "minus": 10,
        "per": 10,
        "default": 10
      }
    },
    {
      "when": {
        "field": "extract_depth",
        "equals": "advanced"
      },
      "addUsd": 0.008
    },
    {
      "when": {
        "field": "extract_depth",
        "equals": "advanced"
      },
      "addUsdPer": {
        "field": "limit",
        "usd": 0.008,
        "minus": 5,
        "per": 5,
        "default": 10
      }
    }
  ]
}
```

Pricing source: [https://docs.tavily.com/documentation/api-credits](https://docs.tavily.com/documentation/api-credits). Pay as you go: $0.008 per credit. Extract 1 credit/5 successful URLs (advanced 2); Map 1 credit/10 pages (instructions 2); Crawl mapping + extraction. Research maximum 110 credits mini, 250 pro. Actual usage.credits required; documented URL/request limits bound reservations..

Price verification date: 2026-11-03.

## Complete input schema


```json
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "The root URL to begin the crawl.",
      "minLength": 1,
      "maxLength": 4096
    },
    "instructions": {
      "type": "string",
      "description": "Natural language instructions for the crawler. When specified, the mapping cost increases to 2 API credits per 10 successful pages instead of 1 API credit per 10 pages.",
      "minLength": 1,
      "maxLength": 4000
    },
    "chunks_per_source": {
      "type": "integer",
      "description": "Chunks are short content snippets (about 800 characters each) pulled directly from the source. Use `chunks_per_source` to define the maximum number of relevant chunks returned per source and to control the `raw_content` length. Chunks will appear in the `raw_content` field as: `<chunk 1> [...] <chunk 2> [...] <chunk 3>`. If a page splits into no more chunks than `chunks_per_source`, its full content is returned. Available only when `instructions` are provided. Must be between 1 and 5.",
      "minimum": 1,
      "maximum": 5,
      "default": 3
    },
    "max_depth": {
      "type": "integer",
      "description": "Max depth of the crawl. Defines how far from the base URL the crawler can explore.",
      "default": 1,
      "minimum": 1,
      "maximum": 5
    },
    "max_breadth": {
      "type": "integer",
      "description": "Max number of links to follow per level of the tree (i.e., per page).",
      "default": 20,
      "minimum": 1,
      "maximum": 100
    },
    "limit": {
      "type": "integer",
      "description": "Total number of links the crawler will process before stopping.",
      "default": 10,
      "minimum": 1,
      "maximum": 100
    },
    "select_paths": {
      "type": "array",
      "description": "Regex patterns to select only URLs with specific path patterns (e.g., `/docs/.*`, `/api/v1.*`).",
      "items": {
        "type": "string"
      }
    },
    "select_domains": {
      "type": "array",
      "description": "Regex patterns to select crawling to specific domains or subdomains (e.g., `^docs\\.example\\.com$`).",
      "items": {
        "type": "string"
      }
    },
    "exclude_paths": {
      "type": "array",
      "description": "Regex patterns to exclude URLs with specific path patterns (e.g., `/private/.*`, `/admin/.*`).",
      "items": {
        "type": "string"
      }
    },
    "exclude_domains": {
      "type": "array",
      "description": "Regex patterns to exclude specific domains or subdomains from crawling (e.g., `^private\\.example\\.com$`).",
      "items": {
        "type": "string"
      }
    },
    "allow_external": {
      "type": "boolean",
      "description": "Whether to include external domain links in the final results list.",
      "default": false
    },
    "include_images": {
      "type": "boolean",
      "description": "Whether to include images in the crawl results.",
      "default": false
    },
    "extract_depth": {
      "type": "string",
      "description": "Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency. `basic` extraction costs 1 credit per 5 successful extractions, while `advanced` extraction costs 2 credits per 5 successful extractions.",
      "enum": [
        "basic",
        "advanced"
      ],
      "default": "basic"
    },
    "format": {
      "type": "string",
      "description": "The format of the extracted web page content. `markdown` returns content in markdown format. `text` returns plain text and may increase latency.",
      "enum": [
        "markdown",
        "text"
      ],
      "default": "markdown"
    },
    "include_favicon": {
      "type": "boolean",
      "description": "Whether to include the favicon URL for each result.",
      "default": false
    },
    "timeout": {
      "type": "number",
      "format": "float",
      "description": "Maximum time in seconds to wait for the crawl operation before timing out. Must be between 10 and 150 seconds.",
      "minimum": 10,
      "maximum": 150,
      "default": 150
    },
    "include_usage": {
      "type": "boolean",
      "const": true,
      "default": true,
      "description": "Credit usage is always requested for actual-cost billing."
    }
  },
  "required": [
    "url"
  ],
  "additionalProperties": false
}
```

## Published example

This is the provider tool’s published example. Replace example values with the user’s requested inputs, respecting the schema; do not execute an example merely to test a connection.


```json
{
  "tool_id": "tavily/crawl",
  "input": {
    "url": "https://example.com",
    "limit": 10
  }
}
```

## Execute and retrieve

Inspect again before execution, explain the price and obtain the required authority. Use MCP run_tool or POST /v1/run with this tool ID, valid input and an idempotency key. Save the returned run ID and follow the actual state. If Agentik answers that the tool is temporarily unavailable, do not retry it blindly; use discovery to propose an alternative.

A partial or uncertain result can still have consumed provider resources. Consult [billing](/docs/billing.md), [idempotency](/docs/idempotency.md) and [asynchronous results](/docs/async.md).

[Provider documentation](https://docs.tavily.com/documentation/api-reference/endpoint/crawl)

[REST inspection](https://api.agentik.cc/v1/tools/tavily/crawl) · [OpenAPI](/openapi.json) · [Connection guide](/connect.md)

---

[HTML page](/providers/tavily/crawl) · [Agent index](/llms.txt)