How to Remove Crawl Delay from robots.txt File in SiteGround Sites?

SiteGround is one of the popular and recommend web hosts by official WordPress.org community. They use custom Site Tools panel which has complete different options than the traditional cPanel hosting. Recently, I noticed a crawl delay was automatically added in the site’s robots.txt file. Even discussing with support does not help to remove the entry as it is forced in to my robots.txt file. So, if you notice crawl delay entry in your SiteGround site, here is how you can remove it.

SiteGround Crawl Delay

By default, SiteGround servers apply 10 seconds of crawl delay for all user agents including Google, Bing, etc. Y can see the crawl delay line appears in robots.txt file when you open the file in browser.

User-agent: *
Disallow:
Crawl-delay: 10

Upload Manual Robots.txt File with Entries

SiteGround adds 10 seconds crawl delay when the site does not have robots.txt file located in the root of the server. On some sites, I also noticed the crawl delay is added even with a blank robots.txt file. So, the solution is to upload a custom robots.txt file with few entries.

• Open a plain text processing app like TextEdit or Notepad on your computer.
• Create a blank file and add your entries. If you do not have anything to block, simply add your XML Sitemap directive like below:

Sitemap: https://yoursite.com/sitemap_index.xml

• If you want to reduce the crawl delay for some reason, the minimum value can be set as 1. The below directive will overwrite the default 10 seconds delay to 1 second.

User-agent: *
Crawl-delay: 1

• Save the file as plain text file with the name as robots.txt.
• Login to your SiteGround hosting account and go to “Site Tools” section for the site that you want to remove the crawl delay.
• Navigate to “Site > File Manager” section.
• Go to public_html folder and click the “Upload” icon.
• Select your robots.txt text file and upload it on your server.

Now, check the file URL (https://yoursite.com/robots.txt) in a browser window and you should not see the crawl delay entry in the file. If you are using WordPress plugins like Yoast SEO, I recommend using their crawl optimization settings. This will automatically add some blocking entries in your robots.txt file thus removing the default crawl delay.

As mentioned, sometimes you may see the crawl delay is dynamically inserted in your existing robots.txt file. In this case, download your current file as a backup. Now, delete the file from File Manager and then re-upload the backup file.

Enable NGINX Direct Delivery

Your robots.txt file will be cached since it is a static file. I have to turn off NGINX Direct Delivery and turn on again in SiteGround account to see the correct robots.txt file in browser. Here is how to disable/enable static caching and check if that can show correct robots.txt file.

• Login to your SiteGround hosting account.
• Go to “Websites” tab and click “Site Tools” button showing below the site that you want to edit.
• When you are in Site Tools, go to “Speed > Caching” section from left sidebar.
• Under “NGINX Direct Delivery” tab, disable or enable the primary and all subdomain items.

Your changes will be automatically saved and open robots.txt file in a new browser tab. Now, the crawl delay entry should disappear and you should only see the custom directives added by you.

Note: If you are using Cloudflare or third-party CDN/caching, make sure to purge the robots.txt file URL to see the latest changes. WordPress users can clear the cache from admin panel if you are using SG Optimizer plugin.

Support May Not Help

When I noticed the crawl delay in robots.txt file all of sudden, I contacted the support through chat. The guy was saying SiteGround does not add any entries in robots.txt which is correct only if you have custom file with existing entries. He raised a ticket after I insisted him and the response confirmed SiteGround sets a crawl delay from server side. But it was also a typical copy paste response with a link to SiteGround KB article without explaining why it was added on a site hosted on SiteGround for the past 7 years with custom robots.txt file.

Final Words

Crawl delay is a good feature to stop bots frequently accessing your server, especially bad bots. However, it does not make sense to apply automatically even on Cloud VPS hosting where sufficient server resources are available or can be auto-scaled. I hope this article helps you to understand if you are breaking your head with crawl delay added in robots.txt file in your SiteGround sites.