Robots.txt Generator
Robots.txt is a text file that webmasters use to tell search engine crawlers which pages on their website should not be crawled. The Robots.txt Generator tool will create a robots.txt file for you that you can upload to your website to tell search engine crawlers which pages on your website you do not want them to crawl.