Forum Moderators: DixonJones

Message Too Old, No Replies

Mediapartners-Google

new crawler from G?

         

keyplyr

11:49 pm on Oct 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




First time I've seen this:

64.68.87.69 - - [15/Oct/2003:06:50:51 -0700] "GET /robots.txt HTTP/1.0" 200 903 "-" "Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)"

mack

11:51 pm on Oct 15, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



generaly thats the bot that spiders your pages containing adsense code on them.

It is used to determine page content so they can deliver targeted adverts.

Mack.

Jenstar

12:09 am on Oct 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you don't run AdSense on that site, it can also be from Opera users who are displaying AdSense ads.

keyplyr

4:50 am on Oct 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Ah... thanks.

<added> So Opera users spoof the Googlebot UA and request robots.txt on people's websites? I always was a bit suspicious of those Opera users! :)

ams_david

6:55 am on Oct 16, 2003 (gmt 0)

10+ Year Member



Opera isn't spoofing (I don't think).

If google ads are enabled in Opera, every time a user goes to a new page, a notice is sent to google with that page's address. If google has no cache of the page, then MediaPartners robot shows up to analyize the page for contextual advertising.

keyplyr

8:26 am on Oct 16, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks for clearing that up ams_david.

Opera is the WAP browser for my mobile phone, but not the web browser for my computer.