SpiderFly journal

Engineering

Design notes for bounded, explainable collection systems.

2 field notes

Newest first

When a web data request needs a browser

A practical way to choose between direct fetch and browser rendering without paying browser cost for every page.

Define scope, depth, and result ownership before a crawl starts so asynchronous collection stays predictable.