SERP guide · DIY first, managed second
How to Scrape Google SERP in 2026 (and When to Stop)
Start with what a home-made Google scraper has to handle, then do the same job as one POST. Before either route, read the terms of the site you plan to collect from: that call is yours to make.
Published · Updated
First: terms and compliance
Search engines publish terms that cover automated access. We do not give legal advice; assess the terms that apply to your use before you collect results, whichever route you choose.
This guide does not show how to get around blocks. The DIY outline below is a sketch to show the moving parts, not a scraper to run.
The DIY pain ledger
Four jobs a self-built scraper takes on. They are described by kind, not by success-rate numbers.
Proxies
Requests from one address are quickly treated as automated, so you end up sourcing, rotating and paying for a proxy pool.
Challenges
Consent screens and challenge pages interrupt sessions; each one needs detection and a fallback path.
Layout changes
The results page markup changes without notice, so selectors that worked last week can return nothing today.
Parser upkeep
Every field you extract (rank, title, url, snippet) is code you maintain, test and fix when the page shifts.
Before and after: the same job in code
# fragile: illustrates the moving parts only
html = fetch_through_proxy(search_url(query, country, lang)) # proxy pool you run
if looks_like_challenge(html): # detection you maintain
retry_later()
rows = parse_results(html) # selectors break when the layout changesimport os, requests
r = requests.post("https://kieapi.com/v1/serp/google/organic",
headers={"Authorization": f"Bearer {os.environ['KEY']}"},
json={"q": "standing desk", "country": "US", "hl": "en"})
if r.status_code == 503:
raise SystemExit("Google SERP endpoint not open yet (503, not charged)")
for row in r.json()["organic"]:
print(row.get("rank"), row["title"], row["url"]){"engine":"google","query":{"q":"standing desk","country":"US"},"status_code":200,
"organic":[{"rank":1,"title":"…","url":"https://…","snippet":"…"}],"organic_count":10}Locale instead of URL tricks
- country picks the market (2–8 characters, default US).
- hl picks the interface language (2–16 characters, default en).
- There is no page parameter: one request returns the parsed first results page.
Launch status for this page
Check engine availability before choosing an endpoint. Google also supports News, Images and asynchronous tasks.
- GoogleGoogle is callable today with a Bearer API key.
SERP searches are charged per successful search at a display rate of 2 Credits per search; failed searches are not charged. Polling task status is free. The current rate is on Pricing.
Frequently asked questions
Is scraping Google allowed?
We do not make legal judgments. Read the terms that apply to your use and decide before you collect results, with or without an API.
What does the managed call return?
organic[], where each row has rank, title, url and snippet (plus display_url), and organic_count with the number of rows.
What if the endpoint is not open yet?
The request returns 503 service_unavailable and is not charged. Retry once the endpoint opens; the launch status is on this page.
Is the Google endpoint open now?
Google is callable today with a Bearer API key. The SERP API as a whole is not GA.
Retire the parser
Keep the query, drop the proxy pool and selectors, and read organic[] as JSON.