Scrapy Web Scraping
Helps with web development, web scraping as part of developing backend services and APIs workflows.
Install on your platform
We auto-selected Claude Code based on this skill’s supported platforms.
Run in terminal (recommended)
claude mcp add scrapy-web-scraping npx -- -y @trustedskills/scrapy-web-scraping
Or manually add to ~/.claude/settings.json
{
"mcpServers": {
"scrapy-web-scraping": {
"command": "npx",
"args": [
"-y",
"@trustedskills/scrapy-web-scraping"
]
}
}
}Requires Claude Code (claude CLI). Run claude --version to verify your install.
About This Skill
What it does
The scrapy-web-scraping skill enables AI agents to extract structured data from websites using Scrapy, a powerful web scraping framework. It supports parsing HTML content, navigating through pages, and extracting specific elements based on selectors like XPath or CSS.
When to use it
- Extract product listings or pricing information from e-commerce sites.
- Gather news articles or blog posts for content aggregation.
- Monitor changes in website content over time for data analysis or SEO purposes.
Key capabilities
- Parsing HTML and XML documents
- Using XPath and CSS selectors for targeted data extraction
- Handling pagination and following links across multiple pages
- Exporting scraped data into formats like JSON, CSV, or databases
Example prompts
- "Scrape all product titles and prices from https://example-store.com."
- "Extract the latest headlines from the technology section of news.example.org."
- "Follow all 'Next Page' links on a blog and save each article's content as JSON."
Tips & gotchas
- Ensure compliance with website terms of service and robots.txt rules to avoid legal issues.
- Use proxies or rate-limiting mechanisms to prevent IP bans during large-scale scraping tasks.
Tags
TrustedSkills Verification
Unlike other registries that point to live repositories, TrustedSkills pins every skill to a verified commit hash. This protects you from malicious updates — what you install today is exactly what was reviewed and verified.
Security Audits
| Gen Agent Trust Hub | Pass |
| Socket | Pass |
| Snyk | Pass |
🌐 Community
Passed automated security scans.