Stop Reaching for a Headless Browser to Scrape Documentation Sites

Every few days someone in a scraping forum asks a version of the same question: “I’m collecting documentation text for an AI tool, but the pages render with JavaScript. What’s the lightest way to get the content?”

The answers are always the same — open DevTools, check the Network tab, find the XHR. That advice is correct, and for documentation sites specifically it’s usually unnecessary work.

Documentation sites are not arbitrary web apps. They’re overwhelmingly built by a handful of static site generators, and those generators leave the content sitting in predictable places. Three routes cover most of what you’ll hit, and none of them need a browser.

Route 1: the JSON is already in the HTML

Next.js-based docs (which includes a large share of company developer portals) embed the full page payload in a __NEXT_DATA__ script tag. It’s in the initial HTML response — no JavaScript execution needed.

import json, re, httpx

html = httpx.get(url, follow_redirects=True).text
m = re.search(r'<script id="__NEXT_DATA__" type="application/json">(.*?)</script>', html, re.S)
if m:
    data = json.loads(m.group(1))
    # content location varies by site; dump the tree once and look around
    print(json.dumps(data["props"]["pageProps"], indent=2)[:2000])

Enter fullscreen mode Exit fullscreen mode

Docusaurus (the other big one) doesn’t use __NEXT_DATA__, but it pre-renders the full article text into the static HTML. A plain httpx.get plus a main or article selector gets you everything. The “JavaScript rendering” you see in the browser is hydration for navigation and search — the prose was already there in the initial response.

Thirty-second check: curl -s <url> | grep -c "some sentence you can see on the page". If that returns 1 or more, the content is in the HTML and you’re done. This one check resolves the majority of “it’s JavaScript-rendered” cases, because people judge from DevTools’ Elements panel — which shows the hydrated DOM, not what the server actually sent.

Route 2: the markdown is in a public repo

Most open-source project documentation is markdown in the same repository as the code, usually under docs/. Scraping the rendered HTML means fetching pages one at a time, parsing them, and stripping navigation chrome you didn’t want. Cloning gets you the clean source in one shot:

git clone --depth 1 --filter=blob:none --sparse https://github.com/org/project
cd project && git sparse-checkout set docs

Enter fullscreen mode Exit fullscreen mode

You get the original markdown — headings intact, code fences intact, no nav sidebars, no cookie banners, no per-page rate limiting. For a RAG pipeline this is strictly better input than parsed HTML, and it’s one request instead of several hundred.

Worth saying plainly: check the license before you ingest. Documentation is frequently licensed separately from the code, and “the repo is public” is not the same as “you may redistribute this.”

Route 3: the site publishes a text endpoint

A growing number of documentation hosts expose plain-text views:

• /llms.txt — an emerging convention where a site publishes a curated, plain-text map of itself specifically for LLM consumption. Fast-moving developer-tool companies have adopted it quickly. Always worth one request.
• /sitemap.xml — not text content, but it gives you the complete URL list without crawling, which means you never have to discover pages by following links.
• ReadTheDocs projects usually offer downloadable HTML and often PDF/ePub builds of the entire docs set from the version menu. One artifact, complete content.

Picking a route in under a minute

Signal Route curl output contains visible page text Parse the static HTML — done __NEXT_DATA__ in the HTML Extract and walk the JSON Public repo with a docs/ directory Sparse-clone the markdown /llms.txt returns 200 Start there ReadTheDocs / GitBook host Look for the download build None of the above Now open DevTools

When you actually do need a browser

Some cases are genuinely dynamic, and it’s worth knowing them so you don’t over-apply the above:

• Docs behind authentication where the session is established by client-side JavaScript.
• Content assembled from several API calls at runtime with no single payload — more common in interactive API explorers than in prose documentation.
• Sites that gate on a JavaScript challenge before serving anything, where the challenge is the point.

For a few hundred pages of prose, though, these are the exception. The default assumption should be that the text is already reachable, and the browser is the fallback.

The part that actually costs you

The reason this matters isn’t purity — it’s that headless browsers change the shape of your project. You go from a script anyone can run to a pipeline with a browser binary, a memory ceiling, per-page startup cost, and a new class of flaky failures that only reproduce sometimes. On a few hundred documentation pages, Route 1 or Route 2 typically finishes before a browser-based run has finished launching.

Check whether the content is already sitting there in plain text. Most of the time, it is.

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.