Crawl
Simulates web crawling to extract data from websites based on provided URLs and extraction rules.
Install on your platform
We auto-selected Claude Code based on this skill’s supported platforms.
Run in terminal (recommended)
claude mcp add crawl npx -- -y @trustedskills/crawl
Or manually add to ~/.claude/settings.json
{
"mcpServers": {
"crawl": {
"command": "npx",
"args": [
"-y",
"@trustedskills/crawl"
]
}
}
}Requires Claude Code (claude CLI). Run claude --version to verify your install.
About This Skill
What it does
The crawl skill enables AI agents to systematically traverse and extract information from websites. It supports structured data retrieval, including text, links, and metadata, while respecting website policies and rate limits.
When to use it
- To gather public data for research or analysis tasks.
- When building knowledge bases or databases that require web content.
- For monitoring changes on a set of websites over time.
- To extract structured information from multiple pages in an organized way.
Key capabilities
- Crawls websites while respecting
robots.txtand rate limits. - Extracts text, links, and metadata from crawled pages.
- Supports recursive crawling to follow internal links.
- Provides clean, structured output for further processing.
Example prompts
- "Crawl the official documentation site of TensorFlow and extract all section headings."
- "Gather recent news articles about AI ethics from the last three months."
- "Extract product details from an e-commerce website's catalog pages."
Tips & gotchas
- Ensure compliance with the target websites' terms of service.
- Avoid overloading servers by adjusting crawl rates and depth limits.
Tags
TrustedSkills Verification
Unlike other registries that point to live repositories, TrustedSkills pins every skill to a verified commit hash. This protects you from malicious updates — what you install today is exactly what was reviewed and verified.
Security Audits
| Gen Agent Trust Hub | Pass |
| Socket | Pass |
| Snyk | Pass |
🌐 Community
Passed automated security scans.