Lots of visits from adsense bot

[mlduclos]

Hi

I see in stats lots of visits from google adsense bot Agent String: Mediapartners-Google/2.1 . Much more than the normal crawler. This is right? I think maybe consume an excessive bandwidth to serve all these visits daily from google. What do you thinkk? Should I send some direction in robots.txt?

[Hobbs]

that would be the online equivalent to shooting yourself in the foot.

[DamonHD]

Hi,

You will generally get as much as one Mediapartners bot visit for each time a user visits a page (I do, anyway).

That's how AdSense works out what to show ads for on each page, especially for very dynamic sites or parameter-dependent pages.

If you shut out the Media-bot (probably against ToS) then you won't get any (decent) ads.

This is an ineluctable cost of doing business: unless your pages are huge or your bandwidth VERY expensive it is certainly paid for by any clicks you get.

Rgds

Damon

[Mistra]

How much bandwidth do you have? Do you always exceed the allocated bandwidth?

Cheapest bandwidth is around US$1 per GB. Does Googlebot consume so much bandwidth at your site?

Which one is better paying extra US$1 - US$5 per month or losing the capability to earn thousand of dollars per month like some folks here?

If bandwidth is too expensive for you then go ahead and ban Googlebot from visiting your site and who know you might actually ban Google from indexing your site as well which is actually good for your competitors. Just joking...

RFOL

[netmeg]

I for one would like to understand better how the bot works though; as I mentioned in another posting last week, one of my clients who has one set of AdSense on his home page was visited by the Mediapartners-Googlebot over 9900 times in 24 hours - and they were indexing both thumbnail pictures on the site and then following the links to the full sized pictures (no, it's not a porn site - they were pictures of cars) That seems like pretty odd behavior for the AdSense bot.

[GoldenHammer]

There are always 1x if not 2x Google bots online and crawing my site. It took about 18GB bandwidth from my site in 809,889 hits last month.

[netmeg]

Yea, but the regular Googlebot won't touch this site with a ten foot pole (that's another issue entirely) - it's JUST the AdSense bot, and it's not only going nuts, it's going into places on the site that have nothing to do with AdSense.

[DamonHD]

Hi,

I which case:

1) Check that it really *IS* the G/MP bot by doing a reverse lookup on the IP. Maybe ask over in the spider-ID forum for help. Block it if it's a fake and maybe tell us all the IP address so we can block it too before it attacks us...

2) Contact Google and ask nicely if they can explain to you why it's hitting some of the items that it is; maybe there's something in your site that's confusing the bot for example. Maybe it's a bug in their bot that they can fix. DO NOT demand in strident tones that they turn it off and stick it up their loopback connection! B^>

Rgds

Damon