Return to applications
Website Crawler
Map site URLs, scrape richer page data, or crawl a bounded website section with Firecrawl through NanoGPT billing.
Request
{
"operation": "map",
"url": "https://docs.firecrawl.dev",
"sitemap": "include",
"limit": 10
}API
curl -sS \
-X POST \
-H "Authorization: Bearer $NANOGPT_API_KEY" \
-H "Content-Type: application/json" \
-d '{"operation":"map","url":"https://docs.firecrawl.dev","sitemap":"include","limit":10}' \
"https://nano-gpt.com/api/v1/firecrawl"Website Crawler for web and API workflows
Map sites, scrape pages, and crawl bounded website sections with Firecrawl through NanoGPT. The same workflow is available from the web app and the /api/v1/firecrawl API endpoint.
Common uses
- Build site maps for research and migrations
- Extract clean Markdown or HTML from a known URL
- Crawl bounded documentation, help, or product sections
Data returned
- Discovered URLs
- Markdown, HTML, links, images, screenshots, and summaries
- Page metadata and crawl status details