Free Robots.txt File Generator Tool

Create a customized robots.txt file using robots.txt generator tool with options for blocking popular search bots, crawl-delay, Sitemap directive and directory restriction.


Robots.txt File Generator Tool


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: This is a relative path to your root domain and must contain a trailing slash "/"
 
 
 
 
 
 
   



Copy and paste the above content into a text file and save the file as 'robots.txt'. Alternatively, you can also download the robots.txt file. After having the file, upload it in your root directory.



RELATED TOOLS



RELATED POSTS


ABOUT US

WebNots is a knowledge sharing platform for webmasters and tech geeks. We have published 2000+ free articles focusing on website building and technology. We share our experience and knowledge through blog articles, demos, eBooks, videos and glossary terms for the benefit of webmasters and tech community.