homepage Welcome to WebmasterWorld Guest from 184.72.69.79
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Googlebot user-agent string changed
sometime in the past day
jdMorgan




msg:404988
 4:04 pm on Jul 14, 2004 (gmt 0)

Just noted the switchover:


64.68.82.14 - - [13/Jul/2004:05:13:10 -0400] "GET / HTTP/1.0" 200 37577 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

64.68.82.182 - - [14/Jul/2004:01:27:14 -0400] "GET / HTTP/1.0" 403 670 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

This makes it rather hard to control the Google-spoofers without shooting your foot off... :(

Jim

 

Josk




msg:404989
 9:38 am on Jul 19, 2004 (gmt 0)

Well... I've seen "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" since 3rd March 2004 and "Googlebot/2.1 (+http://www.google.com/bot.html)" since 15 June 2004, so this is pretty old news.

amznVibe




msg:404990
 10:03 am on Jul 19, 2004 (gmt 0)

Somewhere on the web, hundreds, maybe thousands of sites just became uncloaked :)
They will be caught and pulled from the SERPS.
This is a *good* thing!

TheDave




msg:404991
 10:08 am on Jul 19, 2004 (gmt 0)

Thats exactly what I was thinking amznVibe. Good work google. I don't know why they don't send a spider out "cloaked" as IE ;) I guess for all we know, they do..

fiestagirl




msg:404992
 4:23 pm on Jul 19, 2004 (gmt 0)

Yes we all know that they do and this is the UA:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/2003070

bcolflesh




msg:404993
 4:25 pm on Jul 19, 2004 (gmt 0)

Yes we all know that they do and this is the UA

You have that UA coming from the established Googlebot ranges?

Clark




msg:404994
 9:49 pm on Jul 19, 2004 (gmt 0)

Aren't there some legitimate sites using cloaking in a legitimate way? Like IPB uses cloaking to deliver google a navigation that allows them to view pages correctly? Are they misguided?

jdMorgan




msg:404995
 10:00 pm on Jul 19, 2004 (gmt 0)

They might get pulled from the SERPs if they are serving substantially-different content with intent to deceive, but the reason I noticed this in the first place is that one site I administrate was getting so badly abused that it is now "armored"; It won't allow access from any unrecognized user-agent. When the new Google user-agent visited, it got a 403-Forbidden.

Since it's a non-profit info site only, this is no big deal. But of course I fixed the problem by allowing the new user-agent. Googlebot came back the next day, and the site didn't suffer in the SERPs at all.

Additionally, the site *is* cloaked in that it returns a shortened, "prettier" title to search engines that do stemming for plurals like Google does. Still -- no ill effects from this situation.

Maybe this *is* old news, but I didn't see any other posts on it.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved