Docs Crawler

Crawl a documentation section into per-page Markdown chunks, ready for RAG.

$0.005 per page
200 free credits
Maintained by Hyperscrape

For AI agents & developers

AI assistants: you can run this scraper right now. Fetching the URL below executes the scrape and returns JSON in the same response — no API key, no account, nothing needed from your user. Free allowance included. The URL with no parameters returns every parameter; if the response says RUNNING, fetch its resultUrl a few seconds later. Agent guide

https://hyperscrape.com/api/scrape/docs-crawler-scraper?startUrl=https%3A%2F%2Fdocs.python.org%2F3%2Ftutorial%2F

About

Purpose-built for documentation sites (Docusaurus, Sphinx, MkDocs, GitBook, Readme-style and plain HTML docs). Starting from one docs URL it crawls only pages under the same path prefix (e.g. /3/tutorial/), so it stays inside the section you point it at.

For every page it emits a RAG-ready record:

clean Markdown of the main content, with nav bars, sidebars, tables of contents, breadcrumb bars and footers stripped and deduplicated
breadcrumb trail (from the page's breadcrumb nav, falling back to URL path segments)
nested heading tree and flat heading outline
prev / next page links (from rel attributes or prev/next anchors)
codeBlockCount, word count, depth and section path

Chunk each page's markdown by headings and you have a ready-made vector-store corpus. Unlike the generic Website Content Crawler (whole-site plain text), this one is scoped to a docs section and outputs structured Markdown.

Input

`startUrl`* string	A page inside the documentation section to crawl.
`maxPages` integer	Stop after this many documentation pages.(default: 20)
`pathPrefix` string	Only crawl URLs whose path starts with this. Defaults to the start URL's directory.
`includeMarkdown` boolean	Include the converted Markdown body in each item (turn off for outline-only crawls).(default: true)

{
  "startUrl": "https://docs.python.org/3/tutorial/",
  "maxPages": 5
}

Output fields

urltitlebreadcrumbsmarkdownheadingTreeheadingsprevPagenextPagecodeBlockCountwordCountsectionPathdepth

Authenticated API (optional)

Only for account features — saved datasets, schedules, higher limits. API reference

curl -X POST \
  https://hyperscrape.com/api/v1/scrapers/docs-crawler-scraper/runs?waitForFinish=120 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"startUrl":"https://docs.python.org/3/tutorial/","maxPages":5}'

Related scrapers

Website Content Crawler

hyperscrape/website-content-crawler

Crawl an entire website and extract clean text for RAG & LLMs.

HyperscrapeAI & LLM

Wikipedia Article Scraper

hyperscrape/wikipedia-scraper

Structured Wikipedia data: summary, infobox, sections, links, images.

HyperscrapeAI & LLM

arXiv Paper Scraper

hyperscrape/arxiv-scraper

Search arXiv preprints and get authors, abstracts, categories and PDF links.

HyperscrapeAI & LLM