From HTML to structured data for agents

Acquire before you structure

Fetching and extracting are separate decisions. First obtain a usable page through direct fetch or browser rendering. Then apply a typed extraction contract to the accepted HTML.

This separation makes it possible to improve page acquisition without changing the schema consumed by an agent or data pipeline.

Describe the output you need

A small explicit schema is easier to validate than a broad request for every interesting field. Name each field, choose its source, and decide whether an absent value is acceptable before the extraction runs.

• Prefer stable content fields such as title, summary, published date, and canonical URL.
• Keep optional fields optional when the source does not guarantee them.
• Validate item shape before adding extracted values to agent context.

Extract from supplied HTML

When acquisition is already complete, the extraction endpoint can operate on supplied HTML. That keeps network behavior outside the schema step and makes the transformation easier to reproduce.

curl -X POST "$SPIDER_BASE_URL/v1/extract" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/article","html":"<article>...</article>","extraction":{"type":"json_css","schema":{"name":"article","base_selector":"article","fields":[{"name":"title","selector":"h1","type":"text"}]}}}'

Keep source and shape together

Store the source URL, extraction schema version, request ID, and validated output together. An agent should receive structured data with enough provenance for a person to trace it back to the collected page.