Before You Add a CAPTCHA, Map Every Data Exit

Start with the exit, not the CAPTCHA

When a company discovers large-scale scraping, the instinctive response is often to harden the web page: add a CAPTCHA, deploy a browser challenge, collect more fingerprints, and tune mouse or scroll analysis. Those controls can help, but they are not the first question a serious review should ask.

The first question is: where does the valuable data actually leave the system? A collector wants reliable fields at the lowest sustainable cost. If a mobile API, search endpoint, preview route, GraphQL query, export, static JSON file, legacy version, CDN object, or partner surface returns the same data more cheaply, the fortified page is beside the point.

Inventory every data surface

Begin with the business objects worth protecting—prices, inventory, rankings, reviews, contact details, profiles, order state, commercial leads, and proprietary content. For each object, trace every route that can disclose it, not only the route the current frontend calls.

• Which endpoints support pagination, bulk filters, broad search, or enumerable identifiers?
• Which responses work without authentication, or with a newly created low-trust account?
• Do mobile, partner, preview, share, or legacy APIs enforce the same policy as the web application?
• Can one endpoint return more rows or fields than any product screen displays?
• Do caches, object storage, static assets, or old client versions expose another copy?
• Are permissions enforced by the server, or merely hidden by the interface?

Return less data by design

The safest field is the one an unauthorized client never receives. Responses should include only the properties required for the current product action. Internal identifiers, hidden status, full descriptions, contact fields, or administrative metadata should not ride along because a generic serializer made it convenient.

OWASP's API guidance treats excessive property exposure as an authorization problem, not a frontend presentation choice. Define explicit response schemas, review them by audience, and test that sensitive properties do not appear in list, search, preview, and error responses. Data minimization reduces both scraping value and accidental disclosure.

Authorize every object and operation on the server

A disabled button is not access control. Neither is a route that the interface never links to, an identifier that looks unguessable, or a page that requires login while its backing API does not. Every endpoint that reads an object needs to verify whether the caller may access that object and the specific properties returned.

Authorization must be consistent across web, mobile, GraphQL, batch, and legacy surfaces. Automated tests should exercise adjacent object identifiers, alternate encodings, bulk routes, and role boundaries. Fixing the page while leaving a cheaper API path open simply redirects collection to that path.

Classify routes by business value

Not every endpoint needs the same friction. A public homepage, an individual article, a product search, a seller directory, a complete price history, and a customer export have different consequences. Applying the strictest rule everywhere damages users; applying the loosest rule everywhere exposes the business.

Create route classes with explicit field budgets, page-size limits, rate policies, authentication requirements, and audit retention. Combine per-user, per-account, per-token, and broader network limits so distributed access cannot evade every control while legitimate shared networks remain usable.

Score intent, not a philosophical identity

The useful decision is not whether the caller is literally a person. A person can abuse an export; a bot can execute an approved integration. Risk comes from whether the access is consistent with the product purpose, the caller's authorization, and the expected cost of the route.

A session that lands directly on a high-value API, requests only machine-friendly responses, advances through every page, ignores normal discovery paths, or repeats the same pattern across new accounts may deserve lower trust. Those are pieces of evidence, not proof. Good systems combine them with account history, route sensitivity, network reputation, and verified partner identity.

Respond progressively

Immediate blocking is easy to debug and easy to get wrong. Progressive controls can raise the cost of suspicious collection while preserving a usable path for uncertain cases.

• Observe and log low-confidence anomalies before changing the response.
• Reduce pagination depth, expensive joins, or optional fields on sensitive routes.
• Slow repeated access and require stronger identity or authorization as value increases.
• Challenge only when the route and confidence justify user friction.
• Block high-confidence abuse while keeping an appeal and partner-onboarding path.

Trace leaks and measure collateral damage

High-value datasets can carry batch-specific canaries or watermarking that helps locate redistribution, provided the markers do not mislead users or corrupt consequential decisions. Honeypot routes can reveal mechanical enumeration. The objective is attribution and diagnosis: which surface leaked, which account or credential was used, and when the extraction began.

Measure the other side of the ledger as carefully as blocks. Track challenge completion, page latency, API errors, support contacts, partner failures, search visibility, conversion, and accessibility impact. A policy that stops scraping by making the product unusable is not a successful defense.

Make unauthorized extraction uneconomic

Bot defense is ultimately an operating model. Inventory data exits, minimize responses, enforce authorization, classify routes, correlate evidence, apply graduated controls, trace leaks, and review outcomes as the product changes. CAPTCHA is one possible control near the end of that chain, not the foundation.

The goal is not to prevent every program from approaching the service. It is to make bulk removal of valuable data more expensive than its return while keeping normal users, search engines, approved partners, and transparent automation productive. Locking the front door matters only after you know where every other door leads.