Llms Txt Crawler
This skill uses LLMs to crawl and extract specific text content from websites, saving you time on data gathering.
Install on your platform
We auto-selected Claude Code based on this skill’s supported platforms.
Run in terminal (recommended)
claude mcp add llms-txt-crawler npx -- -y @trustedskills/llms-txt-crawler
Or manually add to ~/.claude/settings.json
{
"mcpServers": {
"llms-txt-crawler": {
"command": "npx",
"args": [
"-y",
"@trustedskills/llms-txt-crawler"
]
}
}
}Requires Claude Code (claude CLI). Run claude --version to verify your install.
About This Skill
What it does
This skill allows AI agents to crawl and extract text from web pages. It can retrieve information from specific URLs or perform broader searches based on keywords. The extracted text can then be used for summarization, analysis, or other downstream tasks within the agent's workflow.
When to use it
- Research: Gather background information on a topic by extracting content from relevant websites.
- Content Aggregation: Collect articles or blog posts related to specific keywords and create a curated list.
- Data Extraction: Pull data points (e.g., product prices, contact details) from online listings.
- Competitive Analysis: Scrape competitor websites for pricing or feature information.
Key capabilities
- Web page crawling
- Text extraction
- URL specification
- Keyword-based searching
Example prompts
- "Extract the text content from
https://www.example.com/article." - "Find articles about 'sustainable energy' and retrieve their main body text."
- “Get the product descriptions for all items listed on this page:
https://shop.example.com/products”
Tips & gotchas
The skill’s effectiveness depends on the target website's structure; complex or heavily dynamic sites may yield incomplete results. Be mindful of robots.txt and rate limits to avoid being blocked by websites.
Tags
TrustedSkills Verification
Unlike other registries that point to live repositories, TrustedSkills pins every skill to a verified commit hash. This protects you from malicious updates — what you install today is exactly what was reviewed and verified.
Security Audits
| Gen Agent Trust Hub | Pass |
| Socket | Pass |
| Snyk | Pass |
🌐 Community
Passed automated security scans.