robots.txt Generator
Create rules to instruct web robots (typically search engine bots) how to crawl your website.
What is a robots.txt file?
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests.