Forum Moderators: martinibuster
I see that Mediapartners bot makes duplicate requests for
lots of my pages. i.e.
66.249.65.137 - - [04/Jun/2005:17:18:00 +0200] "GET /file.html HTTP/1.1" 200 14248 "-" "Mediapartners-Google/2.1"
66.249.65.137 - - [04/Jun/2005:17:18:01 +0200] "GET /file.html HTTP/1.1" 200 14248 "-" "Mediapartners-Google/2.1"
Is it normal?
Regards.
66.249.66.166 - - [04/Jun/2005:23:47:30 +0300] "GET /file.html HTTP/1.1" 200 21065 "-" "Mediapartners-Google/2.1"
66.249.66.166 - - [04/Jun/2005:23:47:31 +0300] "GET /file.html HTTP/1.1" 200 21065 "-" "Mediapartners-Google/2.1"
66.249.66.166 - - [04/Jun/2005:23:47:31 +0300] "GET /file.html HTTP/1.1" 200 21065 "-" "Mediapartners-Google/2.1"
As i can understand, this is meant to be as test for static content/fast changing dynamic content. There can be scraper- and spammerscripts, generating new, random content EVERY TIME. And so mediabot can be sure, that crawled content stable in some degree.
What is wrong here?:)