Firecrawl Scraping

What it does

This skill, Firecrawl Scraping, extracts structured data from individual web pages and converts them into clean, LLM-ready markdown or other formats. It utilizes headless browser automation to handle JavaScript rendering, anti-bot protection measures, and dynamically loaded content. The tool is designed for scraping single pages; multi-page website crawling should use the Apify Website Content Crawler instead.

When to use it

• Scraping a specific article from a blog or documentation site.
• Extracting data from a webpage with heavy JavaScript rendering that standard scrapers struggle with.
• Accessing content behind basic anti-bot protections (e.g., premium news sites like the WSJ or NYT using the "stealth" proxy).
• Generating summaries of web pages.
• Capturing screenshots of webpages.

Key capabilities

• JavaScript Rendering: Handles websites that rely heavily on JavaScript to load content.
• Anti-Bot Protection Handling: Offers proxy modes ("basic," "stealth," and "auto") to bypass basic anti-bot measures.
• Output Formatting: Supports markdown (default), HTML, summary (AI-generated), screenshot, and links extraction.
• Environment Variable Configuration: Uses a .env file for API key management (FIRECRAWL_API_KEY).
• Respects robots.txt: The tool adheres to the rules defined in a website's robots.txt file.

Example prompts

• "Scrape this article and give me the markdown: https://example.com/article"
• "Extract a summary of this WSJ article, using the stealth proxy: https://wsj.com/article"
• "Get a screenshot of this page: https://www.example.com/"

Tips & gotchas

• API Key Required: You must obtain and configure a Firecrawl API key (stored in a .env file) to use the skill. Do not commit your API key directly into code repositories.
• Credit Consumption: Scraping uses credits, with stealth proxy modes potentially consuming more. Monitor your credit balance on the Firecrawl dashboard (https://firecrawl.dev/app).
• JavaScript Rendering Issues: If a page isn't rendering correctly (empty content), it might be due to complex JavaScript; try increasing the timeout duration using --timeout.