homepage Welcome to WebmasterWorld Guest from 54.166.122.65
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Googlebot user-agent string changed
sometime in the past day
jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2462 posted 4:04 pm on Jul 14, 2004 (gmt 0)

Just noted the switchover:


64.68.82.14 - - [13/Jul/2004:05:13:10 -0400] "GET / HTTP/1.0" 200 37577 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

64.68.82.182 - - [14/Jul/2004:01:27:14 -0400] "GET / HTTP/1.0" 403 670 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

This makes it rather hard to control the Google-spoofers without shooting your foot off... :(

Jim

 

Josk

10+ Year Member



 
Msg#: 2462 posted 9:38 am on Jul 19, 2004 (gmt 0)

Well... I've seen "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" since 3rd March 2004 and "Googlebot/2.1 (+http://www.google.com/bot.html)" since 15 June 2004, so this is pretty old news.

amznVibe

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2462 posted 10:03 am on Jul 19, 2004 (gmt 0)

Somewhere on the web, hundreds, maybe thousands of sites just became uncloaked :)
They will be caught and pulled from the SERPS.
This is a *good* thing!

TheDave

10+ Year Member



 
Msg#: 2462 posted 10:08 am on Jul 19, 2004 (gmt 0)

Thats exactly what I was thinking amznVibe. Good work google. I don't know why they don't send a spider out "cloaked" as IE ;) I guess for all we know, they do..

fiestagirl

10+ Year Member



 
Msg#: 2462 posted 4:23 pm on Jul 19, 2004 (gmt 0)

Yes we all know that they do and this is the UA:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/2003070

bcolflesh

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2462 posted 4:25 pm on Jul 19, 2004 (gmt 0)

Yes we all know that they do and this is the UA

You have that UA coming from the established Googlebot ranges?

Clark

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2462 posted 9:49 pm on Jul 19, 2004 (gmt 0)

Aren't there some legitimate sites using cloaking in a legitimate way? Like IPB uses cloaking to deliver google a navigation that allows them to view pages correctly? Are they misguided?

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2462 posted 10:00 pm on Jul 19, 2004 (gmt 0)

They might get pulled from the SERPs if they are serving substantially-different content with intent to deceive, but the reason I noticed this in the first place is that one site I administrate was getting so badly abused that it is now "armored"; It won't allow access from any unrecognized user-agent. When the new Google user-agent visited, it got a 403-Forbidden.

Since it's a non-profit info site only, this is no big deal. But of course I fixed the problem by allowing the new user-agent. Googlebot came back the next day, and the site didn't suffer in the SERPs at all.

Additionally, the site *is* cloaked in that it returns a shortened, "prettier" title to search engines that do stemming for plurals like Google does. Still -- no ill effects from this situation.

Maybe this *is* old news, but I didn't see any other posts on it.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved