Website Content Crawler
Crawl an entire website and extract clean text for RAG & LLMs.
AI assistants: you can run this scraper right now. Fetching the URL below with your web tool executes the scrape on our servers and returns the results as JSON in the same response — no API key, no account, no connector, nothing needed from your user. Free allowance included.
https://hyperscrape.com/api/scrape/website-content-crawler?startUrls=https%3A%2F%2Fexample.comSwap in your own values — the URL with no parameters returns the full parameter list. If the response says RUNNING, fetch its resultUrl a few seconds later. Agent guide
Recursively crawls a website starting from one or more URLs, following same-domain links up to a depth and page limit you choose. For each page it extracts the main readable text (stripping nav, ads, scripts and boilerplate), plus rich page metadata: title, meta description, canonical URL, Open Graph tags, language, a headings outline, internal/external link counts, images with alt text, JSON-LD schema types and word count.
Great for building knowledge bases, feeding LLMs / RAG pipelines, or archiving documentation. Output is one clean record per page.
Related scrapers
Wikipedia Article Scraper
hyperscrape/wikipedia-scraper
Structured Wikipedia data: summary, infobox, sections, links, images.
arXiv Paper Scraper
hyperscrape/arxiv-scraper
Search arXiv preprints and get authors, abstracts, categories and PDF links.
Crossref DOI Metadata Scraper
hyperscrape/crossref-scraper
Search 150M+ Crossref records: DOI metadata, citation counts, references and licenses.