Web To Markdown

What it does

This skill converts web pages to clean Markdown format using a locally installed browser and the web2md tool. It handles JavaScript-rendered pages by leveraging Puppeteer with a Chrome/Chromium-based browser, extracts the main content of the page (using Readability), and then transforms it into Markdown (with Turndown) including cleaned links and optional YAML frontmatter. This is particularly useful for integrating web scraping and content extraction into automated workflows.

When to use it

• You need to convert a webpage with dynamic JavaScript content into a Markdown document.
• You want to automate the process of extracting content from multiple webpages and saving them as Markdown files.
• A webpage requires user interaction (login) before the main content is available, and you need to incorporate that step into the conversion process.
• You require a specific browser profile for authentication or session management when converting web pages.

Key capabilities

• Handles JavaScript-rendered webpages using Puppeteer and Chrome/Chromium browsers.
• Extracts primary content from webpages using Readability.
• Converts extracted content to Markdown format with Turndown.
• Supports optional YAML frontmatter in the output Markdown.
• Provides options for printing output to standard output or saving it as a file or directory.
• Includes interactive mode for handling login walls and user verification.

Example prompts

• "use the skill web-to-markdown https://www.example.com --out ./output.md"
• "use a skill web-to-markdown https://www.anotherwebsite.org --interactive --user-data-dir ./tmp/profile --out ./output/"
• "use the skill web-to-markdown https://www.dynamicpage.net --print"

Tips & gotchas

• Explicit Skill Invocation: This skill requires you to explicitly request it using "use the skill web-to-markdown" or "use a skill web-to-markdown". Otherwise, the agent will prompt for clarification.
• web2md Dependency: The web2md tool must be installed locally. If missing, the agent will provide instructions on how to install it (typically involving cloning and building from source).
• Browser Compatibility: This skill works best with Chromium-family browsers (Chrome, Chromium, Brave, Edge).