When a web data request needs a browser

Start with the fast path

Most public pages do not need a browser. Begin with a direct fetch, inspect the returned content, and reserve browser rendering for pages that actually depend on JavaScript or interaction.

This keeps the first request simple while preserving a clear upgrade path when the page response is incomplete.

curl -X POST "$SPIDER_BASE_URL/v1/fetch" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/article"}'

Read the response signals

A status code alone does not tell you whether a page is useful. A successful response can still contain an empty application shell, a challenge page, or too little primary content.

Treat the content and public audit fields as one decision surface. The useful question is whether the result is ready for the next step in your product.

• Use the normalized content instead of checking raw HTML length alone.
• Inspect the quality and fallback summary before retrying a target.
• Keep request IDs with downstream records so a collection decision can be reviewed later.

Move to browser rendering when the page requires it

Use the browser endpoint when the page needs JavaScript rendering, a controlled wait, or browser state before its content becomes usable. Make that escalation explicit so cost and behavior remain visible to the caller.

curl -X POST "$SPIDER_BASE_URL/v1/browser/scrape" \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/app"}'

Keep the decision evidence

The collection path should be explainable after the request finishes. Store the winning path, fallback reason, quality summary, cost units, and request ID beside the accepted content.

That evidence helps a team tune target policies without turning every unusual page into a one-off integration.