When a web data request needs a browser

Start with the fast path

Suppose your application collects release notes from a public page that you control. The records must contain a title and a link. A server-rendered version already contains those records in its initial HTML; a JavaScript version inserts the same records after the page loads. This is a useful test because the expected content stays constant while the acquisition method changes.

Begin with one URL and the Fetch API. Check whether the required records are present, rather than treating HTTP 200 or a large response body as success. If the initial HTML already contains the records, browser rendering adds work without improving the accepted dataset.

The example below uses Python 3 and the standard library. Set SPIDER_API_BASE to https://kieapi.com, SPIDER_API_KEY to a Team API key, TARGET_URL to your public test page, and JOB_KEY to a unique identifier for this collection attempt. Keep that key when investigating a disconnected request. Requests can consume the Credits shown in your account; this article does not run them automatically.

Build a page with an observable result

Serve this small fixture at TARGET_URL. It has one release note, a stable CSS selector, and an observable acceptance marker. The static version should be available to Fetch. To test rendering, move the article creation into your own page JavaScript and leave the initial main element empty. Keep the final title, link, and selector identical.

A selector appearing is a readiness signal, not proof that every record loaded. For a real feed, choose a selector for the required data and compare the extracted count and fields against the page you can see. Do not use an arbitrary long sleep as the acceptance check.

<main>
  <article class="release-note">
    <h2>Release 1.2</h2>
    <a href="/releases/1.2">Read the release notes</a>
  </article>
</main>

Read the response signals

The script first calls Fetch, then passes its HTML to Extract. Extract works on supplied HTML; it does not fetch the URL itself. The json_css schema selects each .release-note and returns its title and link. The acceptance rule requires a nonempty result, the expected release title, and a link for every row.

A response that fails this rule needs diagnosis. If the initial page is an application shell and the visible records are generated by JavaScript, set ALLOW_BROWSER=1 for an explicit browser attempt. If it is a login page, a challenge, an unexpected redirect, or an upstream failure, investigate that condition before enabling rendering. Missing records alone do not prove JavaScript is the cause.

Move to browser rendering when the page requires it

Run this complete client against the controlled fixture. The browser request waits up to seven seconds for .release-note and applies the same extraction schema. This keeps the comparison focused on acquisition. It does not expose arbitrary JavaScript, cookies, proxy credentials, or browser profile identifiers.

The client makes at most one Fetch and one explicitly enabled Browser scrape request. It saves the returned envelopes and accepted records to collection.json. A timeout stops the client: it is not permission to issue another charged operation with a new identity. Inspect the original run or retry according to the endpoint idempotency contract with the same request key and body.

import json, os
from urllib.request import Request, urlopen

base = os.environ["SPIDER_API_BASE"].rstrip("/")
key = os.environ["SPIDER_API_KEY"]
target = os.environ["TARGET_URL"]
job = os.environ["JOB_KEY"]
schema = {"type": "json_css", "schema": {
    "name": "release_notes", "base_selector": ".release-note",
    "fields": [
        {"name": "title", "selector": "h2", "type": "text"},
        {"name": "path", "selector": "a", "type": "attribute",
         "attribute": "href"}]}}

def post(path, body, operation):
    request = Request(base + path, json.dumps(body).encode(), {
        "Authorization": "Bearer " + key,
        "Content-Type": "application/json",
        "Idempotency-Key": job + "-" + operation}, method="POST")
    with urlopen(request, timeout=90) as response:
        return json.load(response)

def accepted(data):
    rows = data.get("items", [])
    return (not data.get("error") and bool(rows)
            and any(row.get("title") == "Release 1.2" for row in rows)
            and all(row.get("title") and row.get("path") for row in rows))

fetch = post("/v1/fetch", {"url": target}, "fetch")
page = fetch["result"]
html = (page.get("content") or {}).get("html")
structured = {}
if page.get("status") == "completed" and html:
    extracted = post("/v1/extract", {
        "url": page.get("final_url") or target,
        "html": html, "extraction": schema}, "extract")
    structured = extracted["structured"]
browser = None
if not accepted(structured) and os.getenv("ALLOW_BROWSER") == "1":
    browser = post("/v1/browser/scrape", {
        "url": target,
        "browser": {"wait_for_selector": ".release-note",
                    "wait_for_timeout_ms": 7000},
        "extraction": schema}, "browser")
    page = browser["result"]
    structured = page.get("structured") or {}
ok = page.get("status") == "completed" and accepted(structured)
with open("collection.json", "w") as output:
    json.dump({"target": target, "fetch": fetch, "browser": browser,
               "accepted": ok,
               "items": structured.get("items", []) if ok else []}, output, indent=2)
if not ok:
    raise SystemExit("Acceptance failed; inspect collection.json before retrying")
print("Accepted records:", len(structured["items"]))

Compare the observed outcomes

Illustrative results for the controlled fixture: the static page yields one title/link row after Fetch and Extract, so no browser call is needed. The JavaScript version may yield no usable row from the initial HTML, then one row after the browser wait. These are expected test outcomes, not a claim that a live customer page was fetched for this article.

Inspect result.status, result.audit, result.usage, and any result.diagnostics alongside the records. A completed run with an empty structured.items array still fails this application’s acceptance rule. A low_quality or failed page is not accepted even if a partial title happens to match.

Keep the decision evidence

Keep the input URL, final URL, run.id, page status, extraction schema version, accepted rows, audit, and usage together. Fetch and Browser have separate run identities; retaining both lets a reviewer see why rendering was attempted. Extract has no run.id of its own, so attach its result to the acquisition record.

In production, replace the fixture-specific title check with explicit business requirements, such as a canonical release URL, nonempty title, and supported publication-date format. Record a missing date as unknown. Keep raw HTML and public response files in access-controlled storage if the collected pages contain sensitive data.

Start with one representative static page and one representative JavaScript page. Only expand to multiple URLs once the accepted record shape and failure policy are stable. A browser is an acquisition tool; it does not replace downstream validation or make every inaccessible page collectible.

Continue with the implementation

Keep the Fetch API, Browser API, and Extract API references beside this example. Once one page is reliable, design a bounded crawl. For downstream validation, continue with HTML to structured data for agents; for inconsistent sessions, read the whole-session diagnosis.