Tavily Crawl
tavily-ai
Provides web crawling guidance and assistance for software development workflows.
41,264 listings from across the agent ecosystem. None of them have been code-reviewed — check the source before you install.
Looking for what's new? See the repos gaining stars fastest.
11 skills matching “crawl”
tavily-ai
Provides web crawling guidance and assistance for software development workflows.
tavily-ai
Simulates web crawling to extract data from websites based on provided URLs and extraction rules.
agricidaniel
Generates SEO-optimized sitemaps & XML files to improve website crawlability and search engine rankings for agricultural businesses.
agykit
This skill uses LLMs to crawl and extract specific text content from websites, saving you time on data gathering.
crawlio-app
Crawlio-App's audit-site analyzes a website's structure, links, and content for SEO issues and technical errors.
AgriciDaniel
Full website SEO audit with parallel subagent delegation. Crawls up to 500 pages, detects business type, delegates to 6 specialists, generates health score.
AgriciDaniel
Optimize content for AI Overviews (formerly SGE), ChatGPT web search, Perplexity, and other AI-powered search experiences. Generative Engine Optimization (GEO) analysis including brand mention signals, AI crawler accessibility, llms.txt compliance, passage-level citability scoring, and platform-specific optimization.
AgriciDaniel
Analyze existing XML sitemaps or generate new ones with industry templates. Validates format, URLs, and structure.
AgriciDaniel
Technical SEO audit across 8 categories: crawlability, indexability, security, URL structure, mobile, Core Web Vitals, structured data, and JavaScript rendering.
parcadei
Extracts structured data from websites using Firecrawl, bypassing common anti-scraping measures.
brettdavies
Crawl4Ai extracts structured data from websites by intelligently crawling and parsing content – useful for automated data collection & analysis.
Can't find what you need?