The Vibe Father
Playbook
A web scraper is the classic "agent nails it in an afternoon" project — and the classic "why did it break the moment I ran it again" project. Agents write scraper code fluently, but they write it against the page as it looks right now, with brittle selectors and no politeness, and it rots the first time the site changes a class name. The difference between a scraper that works once for a demo and one that runs reliably for months is discipline: resilient selectors, polite behavior, and respect for the legal line. This playbook covers all three, plus the traps that bite everyone.
Step 0 — Check whether you should scrape at all
Before writing a line, check three things: is there an official API or data export that makes scraping unnecessary; what does the site's robots.txt and terms of service say; and are you scraping public data or something behind a login and consent. This isn't legal advice — every jurisdiction and site differs and you should get real advice for anything commercial — but the honest rule is: prefer an official API, respect robots.txt, don't scrape personal data or content behind auth you don't have rights to, and don't hammer a site you don't own. A scraper is easy to build and easy to build irresponsibly.
Step 1 — Resilient selectors, not brittle ones
This is where agents fail silently. Left alone, an agent grabs the first selector that works — a deeply nested CSS path or an auto-generated class — and that selector breaks the instant the site re-renders. Steer it toward selectors that survive change:
- Prefer stable anchors: semantic tags,
idattributes,data-*attributes, and text content that's unlikely to change, over long positional CSS chains. - Match on meaning, not position: "the price near the 'Add to cart' button" is more durable than "the third span in the fourth div."
- Fail loudly: if a selector finds nothing, the scraper should error and log which field broke — not silently write an empty cell. A scraper that quietly returns blanks is worse than one that crashes.
Tell the agent explicitly: choose the most stable selector available, and make every field's absence an error, not a null.
Step 2 — Be polite, or get blocked
An impolite scraper gets your IP banned and puts real load on someone else's server. Build politeness in from the start:
- Rate-limit. A delay between requests, jittered a little. You're not in a race; a request every second or two is plenty for most jobs and keeps you off the radar.
- Respect
robots.txt. Read it and honor the disallowed paths and any crawl-delay it specifies. - Set a real, honest User-Agent that identifies your scraper, rather than impersonating a browser to sneak past. Some sites offer a contact path for scrapers acting in good faith.
- Cache what you've fetched. Don't re-download a page you already have. Caching is politeness and it makes your reruns fast.
Step 3 — Handle the real page, not the happy page
Real scraping is mostly edge cases. Have the agent handle them explicitly:
- Missing fields — some items lack a price, a rating, a description.
- Pagination — and knowing when to stop.
- Rate-limit responses (HTTP 429) and transient errors — back off and retry, don't spin.
- Structure that changes across pages — the detail page differs from the list page.
- JavaScript-rendered content — if the data isn't in the initial HTML, you need a headless browser, which is slower and heavier; check whether a hidden JSON endpoint gives you the data directly first.
The traps
| Trap | The fix | Bites |
|---|---|---|
| Selector breaks silently, writes blanks | Missing field = error, not null | Always |
| No rate limit, gets IP banned | Jittered delay + respect robots.txt | Fast |
| Data is JS-rendered, HTML is empty | Find the JSON endpoint or use headless | Often |
| Reruns re-download everything | Cache fetched pages | Slow |
Step 4 — Verify against reality
The dangerous thing about scrapers is that broken ones still produce output — just wrong output. So verify: spot-check a sample of scraped rows against the live pages by hand, and write a validation pass that flags rows with missing or implausible fields (a price of zero, a date in the future, an empty title). Encode those checks as tests so a site change that breaks your scraper fails loudly — see writing tests that actually catch bugs. A scraper without validation is a machine for generating confident garbage.
Model picks
Scraper logic is squarely value-model territory — Sonnet 5 or GPT-5.3 Codex write the fetch-parse-store loop cleanly, and Gemini Flash's speed makes the "adjust selector, run, check" cycle fast when you're iterating against a real page. Save a stronger model for genuinely gnarly parsing or anti-bot puzzles, and don't overpay for a task that's mostly mechanical. Pick a boring, well-supported HTTP and parsing library the models know cold.
The move
Check for an API first, respect robots.txt and the terms, choose the most stable selectors you can, be polite by default, make every missing field loud, and validate the output against reality. Do that and you have a scraper that survives site changes instead of one that demos once and dies. When the scraper feeds something bigger, the same rigor carries: build a REST API to serve the data, and get a cross-model pass through the AI code review workflow before you rely on it. For more honestly-scoped weekend builds, see things to build with AI agents. In The Vibe Father, a Scout agent can probe a page's structure while a Builder writes the parser — but a single-site scraper ships fine from one terminal.
