How does the Adsense spider work?

[instinct]

Sorry if this has been answered before:

How does the Adsense spider/bot work? Specifically does it crawl your pages the same way the googlebot does by following links? Or, does it just go directly to the pages that have the Adsense code?

The reason I am asking is that I have a huge site with many thousands of pages which I plan on introducing Adsense to. I won't be putting Adsense on every page, just the ones that make sense. Since it takes googlebot quite a while to crawl all my pages can I expect the same from the Adsense bot?

Thanks in advance...

[ken_b]

Normally the mediabot (adsense) visits a page soon after the first time the page is called up after the code has been installed.

That can be very soon, as in minutes, or a few hours.

[Born_User]

Usually it takes longer than a few minutes... but there was definitely one time that the ads were delivered correctly on the first visit to my pages.

I launched a database driven site about a very specific topic, and literally, within a matter of moments, had many, many live web pages up and running about that very topic. As I was testing the launch and opening up the pages on my browser, I saw adsense delivering the proper ads immediately, as though my opening the page somehow prompted the adsense bot to deliver the correct ads.

Other times, it has taken a day or so to work itself out. But that one time where it was immediate was cool.

[instinct]

Thanks Ken_b and Born_User for the replies, that's the news I wanted to hear :-)

[Brett_Tabke]

However, this does *NOT* (despite rumors to the contrary) get you referrals and/or indexed faster. You must still wait for the big update to get any amount of referrals at all.

The two spiders (engine vs media bot) are not cross pollinating.

However, you may trigger a crawl by GBot by having adsense on the page. Although this looks like 1+1=2, they are still separate systems.

[werty]

Months back, I believe in January, we would launch a site and it would rank almost instantly (2-3 weeks). By watching the log files we noticed it was ONLY mediabot that came by and indexed the site. These pages only stuck around around one week in the index and were dropped. So it was like mediabot was acting like freshbot. After this a standard googlebot spider came by and indexed it. Then the whole Brandy and "sandbox" came along and I have not seen this since.

I am assuming they have changed the way things have been indexed since this point, but I thought it was important to state that in the past that I have seen pages that were added via mediabot.

[adfree]

How puts AS the ads into site context then? Happens all the time. Is this something in-house, like matching up the bulk of pages from one domain against each other and run a top level content filter over?

[thaedge]

Ive often wondered how it was able to guess so quickly. I just added a large chunk of pages/content today and even while testing the ads were pretty closely related. I do think that initially it takes hints from possible overall theme of a site - but at most this is a guess in my own case (on 2 sites).

[Bluepixel]

A little offtopic:
Do I'm the only one who always gets hit twice by the mediabot? I always get two requests from two different ips in the same second.

[ogletree]

Even if your site has not been crawled by mediabot G somehow knows what the topic of your site is. They will put ads that match your generic site until you get the media bot spider. You can prove this by completly changing a sites subject and putting AS on it. You will see that you get ads for the old subject matter.

[phaze]

Mediabot also respects robots.txt, so make sure you aren't preventing it from crawling pages it needs to crawl.