Crawler is a piece of software code used to collect the information of webpages available on the internet and index them in the database. Google has plenty of crawlers for collecting different type of information on the web. AdSense crawler one of the Google’s crawlers, used to collect the details of webpage content in order to serve relevant ads on that page. In this article, we will explain more details about AdSense crawler and how it works.
AdSense Crawler
Google uses AdSense crawler for indexing those pages display AdSense advertisements. This helps to serve relevant ads and is different from the web crawlers used for displaying search engine results. Below are some of the crawlers used by Google:
|
Crawler Name (User-agents) |
Purpose |
| Googlebot | Used to index content for showing in Google web search results. This is also the same crawler used for smartphones. |
| Googlebot-Image | Used to index images for showing in Google image search results. |
| Googlebot-News | Used to collect news feed for showing in Google news search results. |
| Googlebot-Video | Used to crawl videos on the web for showing in video search results. |
| Googlebot-Mobile | Used for Google mobile search on feature phones. |
| Mediapartners-Google | Used for indexing web page content for displaying relevant Google AdSense ads. |
Basics of AdSense Crawler
Unlike Googlebot that crawls all the pages on the web, below are some of distinct use Google AdSense crawler:
- AdSense crawlers will only attempt to crawl the pages on which ad codes are placed.
- You can monitor AdSense crawler errors from your AdSense account under “Crawler access” section and work on them. Remember correcting crawler errors for AdSense will not have any impact on the crawler errors shown in Google Search Console since they are from two different bots.
- Google does not allow controlling the frequency of AdSense crawlers. The frequency is automatic and any changed content may take 1-2 weeks for getting indexed.
- https://yoursite.com and https://www.yoursite.com are considered as two different sites and the redirected URLs are also crawled.
- AdSense crawler follows the robots.txt directives similar to all other Google bots.
- You can also provide access to password protected pages on your site for displaying ads. Learn more on this here.
Controlling AdSense Crawler with Robots.txt
Use the below directive in your robots.txt file to block AdSense crawler:
User-agent: Mediapartners-Google
Disallow: /You can also block Googlebot to avoid pages showing on the search results at the same time allows AdSense crawler to show ads on the site.
The topic was full of information and i would love to get more info on how does they(crawler) work and how to get your site approved with adsense as quickly as possible …. it would be very helpful if i can get notified with the next article you write on this niche …
Love the Article Writer.
Appreciated,
Umer Javed
16/06/2024