Basics of Google AdSense Crawler
Crawler is a piece of software code used to collect the information of webpages available on the internet and index them in the database. AdSense crawler is used to collect the details of webpage content in order to serve relevant ads on that page.
AdSense Crawler
Google uses different crawlers for different purposes. AdSense crawler is used for indexing pages for serving relevant ads and is different from the web crawlers used for displaying search engine results. Below are some of the crawlers used by Google:
Crawler Name (User-agents) | Purpose |
| Googlebot | Used to index content for showing in Google web search results. This is also the same crawler used for smartphones. |
| Googlebot-Image | Used to index images for showing in Google image search results. |
| Googlebot-News | Used to collect news feed for showing in Google news search results. |
| Googlebot-Video | Used to crawl videos on the web for showing in video search results. |
| Googlebot-Mobile | Used for Google mobile search on feature phones. |
| Mediapartners-Google | Used for indexing web page content for displaying relevant Google AdSense ads. |
Basics of AdSense Crawler
Below are some of the basics of Google AdSense crawler:
- AdSense crawlers will only attempt to crawl the pages on which ad codes are placed.
- You can monitor AdSense crawler errors from your AdSense account under “Crawler access” section and work on them. Remember correcting crawler errors for AdSense will not have any impact on the crawler errors shown in Google Search Console since they are two different bots.
- Google does not allow controlling the frequency of AdSense crawlers. The frequency is automatic and any changed content may take 1-2 weeks for getting indexed.
- http://yoursite.com and http://www.yoursite.com are considered as two different sites and the redirected URLs are also crawled.
- AdSense crawler follows the robots.txt directives similar to all other Google bots.
- You can also provide access to password protect pages on your site for displaying ads. Learn more on this here.
Controlling AdSense Crawler with Robots.txt
Use the below directive in your robots.txt file to block AdSense crawler:
User-agent: Mediapartners-Google
Disallow: /
You can also block Googlebot to avoid pages showing on the search results at the same time allows AdSense crawler to show ads on the site.
