Spidering by Mediapartners-Google/2.1

this bot is very aggressive; what is it?

[acronym]

I posted about this in the "Search Engine Spider Identification" forum and got no response. I have emailed [e-mail] and gotten no response.

How can I turn down the volume on this bot? Yesterday I got 5700 accesses by all Google bots yesterday, and this particular bot accounted for almost 4800 of those accesses. Getting hit by multiple IPs, often several times per second.

Has anyone seen this UA:

Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)

coming from Google IPs:

64.68.87.66
64.68.87.69
64.68.86.194
64.68.87.28

GoogleGuy, what is this about? What is the Mediapartners bot, what is it doing, and why is it so aggressive?

Mike

PS: Over 2300 accesses so far today.

[acronym]

I heard back from the Google Team, regarding this bot. They claim that it isn't theirs:

"In regards to the "Media Partners" bot, Google does not maintain or operate this robot. The user agent for Google is "Googlebot"."

I wonder if someone is spoofing the Google UA and IP addresses?

But what for? If you spoof an IP address in a request, you can't get the reply back can you? And if not, isn't this basically amounting to a denial of service attack?

Mike

[jomaxx]

That sounds strange to me. Did your server successfully send the pages and log a status 200?

[acronym]

>>Did your server successfully send the pages and log a status 200?

Yes, 200 on all requests. Using HTTP/1.0

Mike

[Powdork]

It was discussed (a little) in this thread
[webmasterworld.com ]

[acronym]

Powdork, thanks, I had searched WebmasterWorld for mediapartner(s) and didn't find that thread. Weird.

So maybe this is a valid Google bot? If so, why doesn't the Google Team answering email at [e-mail] know about it?

Mike

[Powdork]

I tried too so I could link it and couldn't find it either trying mediapartners and media partners. I have found this to be the case often.

[JayC]

So maybe this is a valid Google bot? If so, why doesn't the Google Team answering email at [e-mail] know about it?

If it's associated with the content ads, it's relatively new... so apparently whoever replied to your email didn't recognize it and gave a standard "our UA is googlebot" answer.

Seems like it's probably Google's. After all, if someone were trying to masquerade as googlebot, they wouldn't make up some new user agent! :)

[acronym]

Thanks everyone.

Could someone please explain or point me to a concise description of what is meant by "content ads" with respect to Google?

Is it possible that the Google "mediapartners" bot is being fed by search queries from another search engine, like ask.com? I think that I might be recognizing a pattern in the queries the Google bot is spidering.

Thanks,

Mike

[JayC]

Could someone please explain or point me to a concise description of what is meant by "content ads" with respect to Google?

Acronym, Google is now offering AdWords ads to certain publishers for placement on content pages. Some examples of where you'll find them, according to Google, are:

HowStuffWorks
Blogger
Weather Underground
Knight-Ridder Digital
BURST! Media

This thread [webmasterworld.com] should get you up to speed.

[acronym]

The Google Bots Team emailed me after I asked "are you SURE it's not yours?" and confirmed that this is indeed their bot. They even apologized for denying it was theirs yesterday.

And they said they would slow it down a bit.

It did over 10,000 accesses yesterday and 4800 today.

But they still did not explain what this particular bot is supposed to be doing.

GoogleGuy, can you tell us about this thing?

Mike

[Key_Master]

acronym, do you use fastclick on that site?

[acronym]

Key_Master, yes I do use Fastclick. Why?

Mike

[acronym]

OK, I think I've figured it out.

Fastclick, used on my site to serve ads, is serving ads for Google. When someone does a search on my site (pretty much all my site does), the query is being sent to Google to see if Google has an ad to serve for that keyword. If they do, Fastclick serves the ad.

Part of what Google gets out the deal, other than ad revenue, is a new page to spider. A URL that came from a media partner, hence the name of the bot.

So, Fastclick is providing Google a big fat pool of URLs to spider. This seems to me to be very clever.

Apparently this is so new that the mainstream bot team at Google hadn't heard of it yet, and the team managing the bot didn't understand how aggressively the bot was tearing through websites.

Anyone think this explanation is possible?

Mike

[Key_Master]

[webmasterworld.com...]