Cloudflare is the popular CDN and security service provider available for website owners. The platform offers WAF (Web Application Firewall) and protect your site from malicious bots and attacks. However, the problem comes when Cloudflare blocks legitimate Googlebot and Bingbot. This can quickly become disastrous for your site’s traffic as blocking Googlebot will result in removing the pages from search results. If you find Googlebot is blocked by Cloudflare, here are some of the settings you should check in your account.
How to Find Cloudflare is Blocking Googlebot?
There are multiple ways to understand that you are blocking Googlebot from crawling your site:
First indication is that your traffic will reduce after enabling Cloudflare setup for your site and the pages will start disappearing from Google search result. So, you can immediately understand that something is going wrong if you are monitoring your site’s traffic. Next option is to check the page indexing issues in your Google Search Console account. You should see plenty of 5xx errors indicating the pages are blocked by server side. The website owner also receives email from Google Search Console mentioning the number of pages blocked by server are increasing. You can try to resubmit a page using URL Inspection tool which will fail showing the page can’t be crawled by Googlebot.
Finally, you can also check it from your Cloudflare account by going to “Go to “Security > Events” section. Filter the results based on Google’s IP to view the number of items blocked with reasons. You can check out this official page from Google to get the complete list of latest Googlebot crawler’s IP addresses. Check under “Service” column (if column is not visible, click “Edit columns” button to add) to find the reasons for the blocking.
Click any of the items to view more details like URL path, ASN and the exact category of blocking. As you can see in the below screenshot under “Managed rules” section, it shows “manage definite bots” is the reason for blocking the IP address.
This will give you a clear indication of how many URLs are blocked and which settings in your Cloudflare account caused the blocking.
Unblocking Googlebot in Cloudflare
The problem here is that Google may take long time for re-crawling and reindexing the disappeared pages. It means, you will lose the traffic till the time the pages are discovered again. If you are not fixing the problem immediately, you may also permanently lose the traffic from Google. After analyzing the security events in Cloudflare account, we listed the below things you should check to fix the issue.
1. Disable Bot Mode
The first thing you should disable in Cloudflare is the bot mode. This option is available for all users including free account. Though this is meant to block unknown and malicious bots, Cloudflare may also block Googlebot IP addresses. Go to “Security > Bots” section and turn off the switch against “Bot Fight Mode” option.
2. Super Bot Mode for Premium Accounts
If you are using Pro and other premium plans in Cloudflare, you will have additional Super Bot Fight Mode option. When you are in “Security > Bots” section, click on the “Configure Super Bot Fight Mode” link.
You will see a list of options showing type of bots and the action you want to take for that type. In ideal scenario, Cloudflare should consider Googlebot as a verified bot and allow the access by default. However, if you see Googlebot is blocked in security events then click on the dropdown against “Definitely automated” item and select “Allow”. This will allow any automated bots like Googlebot to access your site without blocking.
3. Check IP Blocking
Many users manually block list of IP addresses in Cloudflare or even block traffic from a specific country completely. If you have manually added IPs in the blocking list, it is a good idea to review the list. Since Google bot has hundreds of IP addresses, there are possibilities you might have mistakenly blocked Google’s IP or ASN. Go to “Security > WAF” section and click on “Tools” tab. Check if there are any items showing as blocked and review them.
If you see, only few specific Google IPs are blocked in security events section, then you can allow access to those specific IPs. In this way, you can continue to use bot fight mode and avoid blocking Googlebot.
4. Disable Firewall Rules
Cloudflare applies set of firewall rules as part of WAF to protect your site. Free account users do not have any option to configure these rules. However, if you are using premium account, then you can go through these rulesets and disable the items that may cause the problem. When you are in “Security > WAF” section, go to “Managed rules” tab and turn off the switch against “Managed rules” option.
If you are using WordPress for you site, scroll down on the page and disable “Cloudflare WordPress” ruleset.
Now, you can check with any crawling tool to confirm if Googlebot can be able to see your content. You can use the URL Inspection tool in Search Console or use other Google tools like Mobile Friendly Tester or Google PageSpeed Insight for this purpose.
Unfortunately, there are too many bad bots crawl a website available online. This makes the task difficult for Cloudflare to precisely identify and block illegitimate bots. As a false positive, Cloudflare may block or challenge search engine bots and real users. If you have setup challenge mode in Cloudflare, real users can enter the captcha code and visit your site. However, search engine bots cannot do that and simply fail to crawl your site. So, avoid playing around with Cloudflare settings like applying bot mode or blocking IP addresses without knowing the impact. After adding your site to Cloudflare, make sure to monitor your site’s traffic to avoid any unforeseen issues.