Check and Fix Index Coverage Issues in Google Search Console

Google is redesigning their popular Webmaster Tools or Search Console website for the past many months. The new Google Search Console UI is still in beta version with no symptom of completion soon. This makes much confusion in using different tools since the options will open with the mix of old and new UI. In earlier version, Google calls it as Crawl Errors and shows all the error related to each site under different categories. But in the new beta version, the crawl error section is migrated to Index Coverage section.

What is Index Coverage?

Google search engine needs to crawl each page on your site and index in the database. Later when the search query matches the keywords on the index, the page search engine will show the page on the results. Google will follow the submitted XML Sitemap and crawl your site, this happens generally on daily basis. Google search engine may find different errors during crawling or indexing process. In order to help site owners to inform these errors and help them to fix, Google show these errors in Search Console. Login to your Google Search Console account and navigate to “Crawl > Crawl Errors’ section. Here you will see all types of errors along with individual URL details.

Click on the ‘Try the new Search Console’ link on the top of the sidebar menu. This will take you to the new beta version. Here you go to ‘Status > Index coverage’ section to see the error details. In this article let us explain how to fix index coverage issues in Google Search Console.

How Do you Know Index Coverage Issue?

Google is changing to more user friendly way of communication. You have multiple ways to know that your site has index coverage issue:

• You will receive an email from Google when index coverage issues are identified. We have noticed, Google sends email only for certain cases and not for all cases.
• Go to ‘All Message’ section in Search Console and check the messages.
• Navigate to ‘Status > Index coverage’ section and get the details.

Index Coverage Issue – Examples

Below are some simple errors you will see in the search console:

• Submitted URL not found (404)
• Submitted URL seems to be Soft 404
• Server error (5xx)
• Submitted URL marked ‘noindex’
• Submitted URL blocked by robots.txt

Note that all index coverage errors are not necessarily to be resolved. For example, you might have blocked a URL or set noindex intentionally.

Viewing Index Coverage Issues

Go to your Google Search Console account and navigate to ‘Status > Index coverage’ section. Ensure ‘All known pages’ is selected from the drop-down and you will see different groups showing numbers.

• Error – these are the problematic URLs from Google’s perspective. These URLs will not show in the search results. So you should focus on resolving the problem under ‘Error’ section first.
• Valid with warning – these URLs have some problem but still indexed by Google.
• Valid – correct and valid URLs properly indexed by Google.
• Excluded – these pages are not indexed and intentionally excluded using ‘noindex’ tag.

Click on ‘Error’ tab to see the graphical representation of the error count on daily basis. The graph shows the details for the past three months.

Fixing Index Coverage Issues

Scroll down to see the list of errors along with error status and reason.

Let us click on the first problem – ‘Submitted URL marked ‘noindex’. You will again see the details, graph and example list of URLs covered under the error.

Options for Checking Error Pages

Google offers multiple options to check the index coverage error pages. Just click on one of the Error URL to see lot of options appear in the right sidebar like below.

Inspect URL

Click on it to start inspecting the URL. You will get the result like below showing the detailed error, crawl date and index status. As the name indicates, this option is only to inspect the status of the URL.

Test Robots.txt Blocking

Use this option to check when you see robots.txt blocked error. Google will show the live robots.txt file on your site. Click on ‘Test’ button to check whether the URL is allowed in robots.txt file. If allowed, you will see the ‘Test’ button is now changed to ‘Allowed’.

If you have mistakenly blocked the URL in robots.txt file then fix the robots.txt entries.

Fetch as Google

Use fetch as Google option to check how Google view your page when crawling. If you see any error in fetching, fix that error.

View as Search Result

Click on this option to view the search result of your error URL. This will not work for the error pages as Google didn’t index those error pages.

Submit to Index

This option sometimes allows you to submit URL directly but most of times show to use URL Inspection tool. Since, URL inspection is already the first option we don’t understand the purpose of including ‘Submit to Index’ option again.

Validating and Fixing the Errors

Once you understand the issue then try to fix it. The fix really depends on the error message.

• 5xx errors – check your server configuration or discuss with your hosting company.
• 404 not found – try to set 301 redirect for the deleted or URL changed pages and use Fetch as Google to resubmit to index.
• Noindex error – check whether you have set this intentionally otherwise remove the noindex meta tag from the page.
• Blocked by robots.txt – check robots.txt file entries and remove the blocking.

After fixing the URLs come back to Index Coverage section and click on ‘Validate Fix’ button.

Google will start validating your fix and show the status as ‘Validation started’ with the date of start.

Once Google completes the validation, you can see the status as ‘Passed’ if the validation is successful. Till the time of validation, you can see the ‘Validation’ status under ‘Status’ section also.

After successful fix of the error, you will also receive a message like below indicating you have successfully fixed the error.