Robots.txt file tells the search engines which pages in your site are to be indexed and which are not. Baidu webmaster tools offer two options under Robots section.
- One is to check whether your site has any existing robots.txt file and display it in details.
- Other is to create a free robots.txt file for your site.
The Robots option is available under Web Analytics section of Baidu webmaster tools account as shown in the below picture. You also can access this option here without registering for webmaster tools account.
Detect and Display Robots.txt
Select the “Detect Robots.txt” tab, enter your site URL and click on Detect button. You can see the detailed analysis of the robots.txt file if detected in your site.
You need to have a webmaster tools account in order to update the robots.txt file in your site. Note Baidu webmaster tools can only detect up to 48k file size, hence ensure your robots.txt file is not too large.
Generate Customized Robots.txt File
Select the “Generate robots.txt” tab or click on the “Generate” button in the previous step to create your own robots.txt file for your site. Enter the following details and click on Create button:
- User Agent
You can choose Baidu Spider, All and Other option. If you choose other option then you need to enter the name of the user agent.
This indicates whether you wanted the pages to be crawled or not crawled.
Here you need to mention the directory path for the search engines to crawl. This should start with / and merely mentioning / indicates the root directory of your site. Baidu supports the directory length of up to 255 characters, hence ensure your directory is not too long.
Robots.txt file will be generated and shown in the box which you can upload in your site’s directory for controlling the search engines crawling your pages.