How to Provide AdSense Crawler Access to Password Protected Pages?

Crawler is a piece of software code Google uses to crawl the web and index pages in the database. AdSense crawler check the content of your webpages in order to serve better and relevant ads related to your content. Every AdSense publisher should know the basics of AdSense crawler in order to generate more revenue and understand how to resolve certain AdSense issues.

Here is an article explaining the details about AdSense crawler and the enabling crawler access for password protected pages.

• Google uses different crawler (also called as spider or bot) for different purposes. For example, the crawler used for the purpose of web search, mobile search, Google webmaster tools and AdSense are completely different. The crawling error you see in your webmaster tools account is nothing to do with the AdSense crawler error you see in your AdSense account. This means resolving AdSense crawler error will not resolve the errors shown in your webmaster tools account. But Google share the cache between crawlers in order to avoid each crawler trying to access the same content separately and increase the load on your site.
• AdSense crawler updates the crawling information on weekly basis. Currently there is no option provided for publishers to increase or reduce the frequency of crawling.
• AdSense crawler treats each domain and sub-domain as a different site. Example.com and www.example.com are considered as two different sites.
• AdSense crawler accesses only the pages displaying AdSense ads and can be blocked by using a robots.txt file. In case of redirects, the original page is accessed first to ensure the redirect is enabled and hence the access log will still show the original page’s details though the ads are displayed on the redirected pages.

Displaying AdSense Ads on Password Protected Pages

It is common to protect the pages with a password to prevent unauthorized access. For example membership content with login access is generally protected by a user id and password. AdSense crawler can’t access the content behind the password protected pages and hence ads will not be displayed on those pages though AdSense ad codes are placed. To overcome this situation webmasters need to provide access to AdSense crawler so that content can be indexed and relevant ads will be displayed.

• Login to your AdSense account and click on the gear icon to access “Settings”.
• Navigate to “Crawler access” section under “Access and authorization” tab.

In order to provide access to AdSense crawler you should have added and verified your site in Google webmaster tools account. Once you have verified your account click on Add a login button as shown in the below picture.

You will see a popup screen and asking to fill up the details. Assume you have a paid membership section in your site with the following structure:

Enter the details as below:

• Restricted Directory or URL – example: yoursite.com/paidmembers
• Login URL – example: yoursite.com/paidmembers/login.php

Note: Whenever there is a password protection, the login page and a protected content page are two different pages.

• Login method – Post
• Parameters – username and password

• Click on the “Test login” button to test whether AdSense crawler can successfully access your protected pages.
• Click on “Add login” to provide final access.

Now that AdSense crawler can crawl and index the content behind the login page and serve relevant ads to visitors.