Create a customized robots.txt file using robots.txt generator tool with options for blocking popular search bots, crawl-delay, Sitemap directive and directory restriction.
Robots.txt is a simple text file that instructs web crawlers what are the content allowed to be crawled and indexed for public. The file should be uploaded in the root directory of your website (generally "/public_html/"). And the search engines will look your site's root directory for the presence of robots.txt file. Refer our separate article on robots.txt to learn complete details of robots.txt file.
Here is the explanation of values you can provide in the tool to generate robots.txt entries:
Choose from the dropdown that you want to allow or block all robots from crawling your site. Good bots like Google and Bing follow this directive set in robots.txt file, but bad bots do not follow this. You should find such bad robots by looking into your server's log and block them using .htaccess directive.
Crawl delay is the time in seconds between crawling of search robots which is used to control the aggressive bots from slowing down your server.
Most of the shared hosting companies need crawl delay of at least 10 seconds in order to protect the server from aggressive bots. If you have managed, VPS or dedicated server then choose the value as "Default - No Delay. Remember, choosing the value "20 Seconds" will allow all the crawlers (that follow this directive) to index 4320 pages per day. This should not be a problem for smaller sites while bigger site owners can leave this field as default.
Similar to robots.txt, by default Sitemap.xml file also should be available in root directory of your site. Search engines will look for the XML Sitemap in the root directory and crawl the content accordingly. If your Sitemap is located in site's root then leave this field blank.
If you have the Sitemap located in different directory other than root, then enter the complete XML Sitemap URL to inform search engine crawlers where your Sitemap file is located. Regardless of the location of XML Sitemap, ensure to submit the Sitemap in webmaster tools account of Google, Bing, Yandex and Baidu. Here is a search engines Sitemap submission guide for your reference.
You can leave this value same as the field 1 by selecting "Same as Default". Or else select the allow or disallow value for individual search engine robots. This directive will be appended to the default control and will be followed only by that search engines. For example, you can select allow for default value and disallow only for Baidu spider. This will allow all search bots except Baidu.
We have given the options for most popular search engines like Google, Bing / MSN, Yahoo!, Baidu, Yandex, Ask/Teoma and Alexa/Wayback. You can refer the complete user agent list and choose additional bots for blocking.
If you want to restrict specific directories then enter the directory name with the trailing slash. For example, if you want to disallow "yoursite.com/admin/" directory then enter "/admin/" in this field. The tool will allow you to add up to six directories but you can add more directly in the robots.file before uploading to server.
Once you selected the required values, click on the "Create Robots.txt" button to generate the Robots.txt file entries in the text box.
If you made any mistake or wanted to reset the tool to initial values then click on the "Clear" button. This will remove all generated entires from the text box.
The generated entries for your robots.txt file can be copied from this text box.
Below are some of the examples of robots.txt entries created using this tool:
Example1:
Selection:
Default: Allow
Search Robots: Baidu – Disallow
Output:
User-agent: baiduspider
Disallow: /
User-agent: *
Disallow:
Example2:
Selection:
Default: Disallow
Output:
User-agent: *
Disallow: /
Example3:
Selection:
Default: Allow Crawl Delay: 10 seconds
Sitemap: https://www.example.com/sitemap-folder/sitemap.xml
Search Robots: Google – Disallow Restrict Directory: /admin/
Output:
User-agent: Googlebot
Disallow: /
User-agent: *
Disallow:
Crawl-delay: 10
Disallow: /admin/
Sitemap: https://www.example.com/sitemap-folder/sitemap.xml
First copy all the entries generated by the robots.txt generator tool. Open a text editor like Notepad / TextEdit and paste the content. Save the file in "UTF-8" format with the name "robots.txt".
Once you have uploaded the robots.txt file, it should be accessed through the web browser like normal webpage. Open your favorite browser and enter the URL like "yoursite.com/robots.txt" and you should see the file displayed like below.
WebNots is a knowledge sharing platform for webmasters and tech geeks. We have published 3000+ free articles focusing on website building and technology. We share our experience through blog articles, demos, eBooks, videos and glossary terms for the benefit of webmasters and tech community.
All Rights Reserved © 2023 WebNots