How to Generate Robots.txt File from Baidu Webmaster Tools?

Robots.txt file tells the search engines which pages in your site are to be indexed and which are not. Baidu webmaster tools offer two options under Robots section.

  • One is to check whether your site has any existing robots.txt file and display it in details.
  • Other is to create a free robots.txt file for your site.

The Robots option is available under Web Analytics section of Baidu webmaster tools account as shown in the below picture. You also can access this option here without registering for webmaster tools account.

Robots.txt Options in Baidu Webmaster Tools
Robots.txt Options in Baidu Webmaster Tools

Detect and Display Robots.txt

Select the “Detect Robots.txt” tab, enter your site URL and click on Detect button. You can see the detailed analysis of the robots.txt file if detected in your site.

Detect and Display Robots.txt File
Detect and Display Robots.txt File

You need to have a webmaster tools account in order to update the robots.txt file in your site. Note Baidu webmaster tools can only detect up to 48k file size, hence ensure your robots.txt file is not too large.

Generate Customized Robots.txt File

Select the “Generate robots.txt” tab or click on the “Generate” button in the previous step to create your own robots.txt file for your site. Enter the following details and click on Create button:

  • User Agent

You can choose Baidu Spider, All and Other option. If you choose other option then you need to enter the name of the user agent.

  • Status

This indicates whether you wanted the pages to be crawled or not crawled.

  • Path

Here you need to mention the directory path for the search engines to crawl. This should start with / and merely mentioning / indicates the root directory of your site. Baidu supports the directory length of up to 255 characters, hence ensure your directory is not too long.

Robots.txt file will be generated and shown in the box which you can upload in your site’s directory for controlling the search engines crawling your pages.

Generate Robots.txt File
Generate Robots.txt File

Editorial Staff at WebNots are team members who love to build websites and share the learning with webmasters community. Follow us on Facebook and Twitter.

Leave a Comment