Adsense nearly quadruples spidering

Added adsense and googlebot has nearly quadrupled spidering.

[thecoalman]

I have a small site, main page is PR3. The forum however is quite popular and has a PR4, about 1200 possible URL's to spider and googlebot according to webmastertools was spidering about 125+ pages a day. I added adsense a few weeks ago and too my suprise on my last check of the crawl rate it has spiked to almost 600 a day since then and is holding steady. No other changes recently to speak of. Just thought some of you might find that of interest.

[Vimes]

Hi,
sounds like the stats software you are using isn't aware that there is an Adsense bot that uses Googlebot in its useragent.

"Adsbot-Google: crawls pages to measure AdWords landing page quality. We only use this bot if you use Google AdWords to advertise your site. Find out more about this bot and how to block it from portions of your site."

theres more info on it here:

[adwords.google.com...]

Vimes.

[thecoalman]

These stats are not server stats but from Google webmastertools, unless its counting the adsense bot in webmastertools?

Upon futher review let me say instead of 600 it's averging about 500 a day. It spiked at 600 one day.

[ansible]

I think what you're seeing is this: Each time a user loads up a page on your site with Adsense ads on it, Googlebot accesses the same page to look at its content, so it can deliver targeted ads (based on the text for that page).

If you look at your raw web server logs, you'll see that the increase in number of Googlebot accessed pages occurs with the exact same user page requests at almost the same time. And you'll also notice that the bot access increase is only accessing the pages that have Adsense on them.

[thecoalman]

I'm under the assumption the Google Bot and the Adsense bot are two different bots? I know I had to specifically allow the adsense bot on all pages that were denied to Google Bot.

Having said that adding adsense actually helps increase spidering then? As I mentioned there is only about 1200 pages accessible by Gooogle Bot, technically it would spider my entire site every 3 days or so. That's assuming it's not pullling up the same popular pages over and over.

[Vimes]

I think you might be right there coalman,

i miss read the first post, didn't mean to confuse the issue.

Vimes.