The crawl control feature in Bing webmaster tools allows webmasters to control the rate at which Bingbot crawls your website. Crawl controlling is required to limit the activities of search engine spiders like Bingbot and Googlebot when there are visitors on your site to manage the bandwidth suitably.
Since visitors on the site are to be given more priority it is obvious that the search engine activities at that time are to be restricted. You can inform Bingbot to crawl faster or slower any point of time during a day according to the visitor’s pattern on your site.
Setting a Bingbot Crawl Pattern
Login to your Bing webmaster tools account and select the site you want to set the crawl control pattern. You must have added and verified you site in your account in order to use this feature. The “Crawl Control” option is available under “Configure My Site” tab.
Setting Preset Pattern
You can see the preset options available under the drop down menu of the question “When do you receive the most traffic to this site for your local time of day?”. “All Day” is selected by default and you can choose any time frame based on the visitors pattern of your site. The pattern automatically changes as per the time frame you selected.
Bingbot Crawl Control – Preset
Save your settings after you selected the suitable crawl control pattern.
Setting Custom Crawl Pattern
If the preset time frames are not suiting your requirement then select the “Custom” option available in the drop down to set your custom crawl control pattern. Hold the left button of your mouse and move over the blue color bricks to build your own crawl pattern as shown in the below picture:
Bingbot Crawl Control – Custom
By default 5 blue bricks are selected for anytime and more the blue bricks faster the crawling speed and lesser the blue bricks slower the crawling speed of Bingbot. Save your settings after you selected the suitable crawl control pattern.
You also can set a crawl delay in your robots.txt file. The crawl rate set in your Bing webmaster tools will be superseded with the information available in your robots.txt.
It is important that the webmasters should first ensure the site slowness is due to Bingbot activity or not. Use verify Bingbot tool to check the IP address shown in your server log file belongs to Bingbot and then set the crawl rate accordingly based on the Bingbot crawl pattern and visitors pattern.
1 Comment
Leave your reply.