Engineering
Design notes for bounded, explainable collection systems.
7 field notes
Newest first
Bot Detection Judges the Whole Session, Not a Single Fingerprint
Modern defenses correlate network identity, navigation, interaction, resource timing, and session history instead of trusting one browser fingerprint.
Browser Fingerprinting Is a Consistency Audit, Not a Checklist
Canvas, WebGL, audio, clean realms, WebAssembly, storage, and automation traces become meaningful when they are evaluated as one device story.
Bot Detection Starts Before HTTP
TCP, TLS, HTTP/2, headers, and QUIC each reveal part of a client implementation—and contradictions across layers are stronger than any one fingerprint.
Before You Add a CAPTCHA, Map Every Data Exit
The cheapest path to valuable data is often an under-protected API, search route, export, preview, or legacy surface—not the hardened web page.
Modern Bot Defense Is an Identity Consistency Problem
Why modern defenses correlate network, browser, device, behavior, and reputation signals—and what responsible web data teams should learn from that design.
POST /v1/browser/scrapeWhen a web data request needs a browser
A practical way to choose between direct fetch and browser rendering without paying browser cost for every page.
POST /v1/crawlDesigning bounded crawls for production
Define scope, depth, and result ownership before a crawl starts so asynchronous collection stays predictable.