🤖 Robots.txt Generator
⚡ Quick Preset
🎯 Select crawler
Press Enter to add a custom crawler
📜 Crawling rules
🗺️ Sitemap
⚙️ Advanced Options
Recommended time between crawler requests
Specify the primary domain name (unique to Yandex)
📄 Generated robots.txt
robots.txt
💡 robots.txt Description
- User-agent : Specify the crawler to which the rule applies
- Disallow : Prohibited crawling paths
- Allow : Paths allowed to be crawled (higher priority)
- Sitemap : Site map location
- The file should be placed in the website root directory: example.com/robots.txt
⚠️ Notes
robots.txt is just a suggestion and may be ignored by malicious crawlers. For sensitive content, server-side access control should be used.