🤖 Robots.txt Generator

⚡ Quick Preset
🎯 Select crawler
Press Enter to add a custom crawler
📜 Crawling rules
🗺️ Sitemap
⚙️ Advanced Options
Recommended time between crawler requests
Specify the primary domain name (unique to Yandex)
📄 Generated robots.txt
robots.txt

💡 robots.txt Description

  • User-agent : Specify the crawler to which the rule applies
  • Disallow : Prohibited crawling paths
  • Allow : Paths allowed to be crawled (higher priority)
  • Sitemap : Site map location
  • The file should be placed in the website root directory: example.com/robots.txt

⚠️ Notes

robots.txt is just a suggestion and may be ignored by malicious crawlers. For sensitive content, server-side access control should be used.