Designing bounded crawls for production

Define the boundary before the first request

Consider a documentation archive with a landing page, several version pages, and a few links to account settings and external reference sites. Your application needs a small snapshot of documentation text, not every reachable URL. Define the seed, allowed hosts, depth, maximum pages, and acceptance rule before creating a job.

For the worked example, use a public documentation fixture that you control. Set DOCS_URL to its landing page. Choose max_depth=1 and max_pages=10 for a first run; disable sitemap discovery so this experiment follows only the chosen seeds and their bounded discovery path. These values are ceilings, not a promise that ten useful pages exist.

Use Python 3 with the standard library. Set SPIDER_API_BASE=https://kieapi.com, SPIDER_API_KEY to your Team API key, DOCS_URL to your test page, and JOB_KEY to a new collection identifier. Keep the same key and body when resolving an uncertain creation outcome. Review account pricing before execution; the script creates a real job when you run it.

Keep scope explicit

same_hostname follows the exact hostname. same_domain also permits hosts sharing the registrable domain, including subdomains. same_origin requires the same scheme, hostname, and effective port. For this example, same_origin keeps HTTPS documentation on its original origin and excludes a linked external reference site.

The public Crawl request currently has no include/exclude path-pattern fields. Do not add invented fields such as include_patterns or assume that a /docs seed restricts all discovery to /docs. Scope is a host/origin boundary; another path on the same origin can still be eligible. If you need an exact URL set, provide those approved seeds with max_depth=0 and keep sitemap discovery off.

A useful fixture links from /docs to two documentation pages, one external URL, and one same-origin non-documentation page. Check which URLs appear in results and why. This exposes the difference between origin scope and path scope before you apply the job to a larger site.

Treat the crawl as a job

Create the job once. A successful creation returns 202 with crawl.id and run.id; neither means collection has finished. Save the complete creation response before polling so a process restart can resume the existing job instead of creating a duplicate.

This client checks status every five seconds for up to five minutes. queued, running, and paused remain nonterminal. completed, failed, low_quality, budget_exhausted, and expired stop the polling loop. An unfamiliar status stops with an error so a changed contract cannot silently become an infinite loop.

import json, os, time
from pathlib import Path
from urllib.parse import urlencode, quote
from urllib.request import Request, urlopen

base = os.environ["SPIDER_API_BASE"].rstrip("/")
key = os.environ["SPIDER_API_KEY"]
job_key = os.environ["JOB_KEY"]

def call(path, body=None):
    headers = {"Authorization": "Bearer " + key}
    data = None
    if body is not None:
        data = json.dumps(body).encode()
        headers.update({"Content-Type": "application/json",
                        "Idempotency-Key": job_key})
    with urlopen(Request(base + path, data, headers), timeout=90) as response:
        return json.load(response)

created = call("/v1/crawl", {
    "urls": [os.environ["DOCS_URL"]], "scope": "same_origin",
    "max_depth": 1, "max_pages": 10, "discover_sitemap": False})
Path("crawl-created.json").write_text(json.dumps(created, indent=2))
crawl_id = created["crawl"]["id"]
path = "/v1/crawl/" + quote(crawl_id, safe="")
active = {"queued", "running", "paused"}
terminal = {"completed", "failed", "low_quality", "budget_exhausted", "expired"}
deadline = time.monotonic() + 300
while True:
    status = call(path)
    Path("crawl-status.json").write_text(json.dumps(status, indent=2))
    state = status["crawl"]["status"]
    if state in terminal:
        break
    if state not in active:
        raise SystemExit("Unknown crawl state: " + state)
    if time.monotonic() >= deadline:
        raise SystemExit("Polling deadline reached; resume this crawl ID later")
    time.sleep(5)

cursor = None
seen = set()
with open("crawl-pages.jsonl", "w") as output:
    while True:
        params = {"limit": 100}
        if cursor:
            params["cursor"] = cursor
        page = call(path + "/results?" + urlencode(params))
        for item in page["items"]:
            content = item.get("content") or {}
            accepted = (item.get("status") == "completed"
                        and bool((content.get("markdown") or "").strip()))
            output.write(json.dumps({"crawl_id": crawl_id,
                "accepted": accepted, "page": item}) + "\n")
        cursor = page.get("next_cursor")
        if not cursor:
            break
        if cursor in seen:
            raise SystemExit("Repeated result cursor; stop and inspect")
        seen.add(cursor)
print("Crawl", crawl_id, "ended with", state)

Read a partial outcome without losing useful pages

Illustrative outcome: the job reaches a terminal state, six pages have nonempty completed Markdown, one page fails, and another contains low-quality content. Your application should keep the six accepted records and preserve the failed records for review. It should not report every discovered or processed page as delivered content.

Read crawl.stop_reason, crawl.queue_counts, and crawl.diagnostics when present. block_reasons and retry.categories summarize observed failures; they are evidence for the next decision, not instructions to resubmit the whole crawl. A budget_exhausted result can still contain useful pages. A completed job still needs page-level acceptance.

The sample accepts nonempty completed Markdown only to demonstrate the lifecycle. Before production, add your own document rules: expected language, title, canonical URL, sufficient main content, and exclusion of account or search pages. Keep rejected pages in the evidence file so a missing document is visible.

Handle failures deliberately

HTTP errors raise an exception in the example. In a production client, record the HTTP status and sanitized public error.code, error.message, and error.retryable. Resolve 401/403 authentication or access failures; correct validation failures; honor Retry-After when returned for throttling. Bound retries and avoid automatically repeating job creation with a fresh key.

A connection failure after POST leaves creation uncertain. Retain JOB_KEY and the request body, and resolve the original operation through the documented idempotency behavior. If crawl-created.json exists, use its crawl.id to resume GET polling. A client polling deadline does not stop or cancel a server-side job.

Read result pages until next_cursor is absent or null; do not stop after the first batch or infer completion from a short batch. If a download is needed, use only an artifact URL returned for that page, including its parent crawl query parameter. Do not construct artifact keys or assume retained files remain available forever.

Retain page-level context

The output files separate creation, latest job status, and page records. Keep them together with the request body and your acceptance-rule version. Each page record retains its URL, final URL when returned, content, audit, usage, and diagnostics. The crawl ID ties that record back to the bounded collection request.

For a durable application, upsert by crawl ID and canonical or final page URL, record the collection timestamp separately from the source publication date, and preserve missing source dates as unknown. If you consume results before the job becomes terminal, refresh from the beginning afterward and deduplicate rather than assuming a temporary end cursor means the job is done.

Before increasing max_pages or enabling sitemap discovery, review the fixture results: no external origin, no unexplained accepted page, a finite poll loop, complete cursor traversal, and traceable rejected pages. Scale the bounds only when those checks match the dataset your application actually needs.

Continue with the implementation

The Crawl API reference defines request limits, statuses, pagination, and artifact access. Validate the first page with the browser decision walkthrough, then use HTML to structured data for agents and the Extract API to turn accepted content into typed records.