homepage Welcome to WebmasterWorld Guest from 54.198.224.121
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Googlebot Spoofing
How do you tell the real googlebot from spoofers?
teaperson




msg:1526234
 6:03 pm on Dec 20, 2004 (gmt 0)

My traffic reports show a lot of bandwidth going to various spiders claiming to be googlebot in their user-agent field. I'm pretty confident that the actual Googlebot wouldn't forget to close the parenthesis on the URL. Does anyone know definitively what Googlebot will call itself?

And is there any point in trying to ban the fake googlebots, or will those spiders just learn to spoof better?

 

benevolent001




msg:1526235
 6:08 pm on Dec 20, 2004 (gmt 0)

its like

Agent: Googlebot/2.1 (+http://www.google.com/bot.html)

Yahoo! Slurp; [help.yahoo.com...]

Agent: msnbot/0.3 (+http://search.msn.com/msnbot.htm)

ogletree




msg:1526236
 6:29 pm on Dec 20, 2004 (gmt 0)

Maybe you could check all the server vars. Like if you see a gbot using a windows os or mac. This can be done in htaccess or php.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved