Welcome to WebmasterWorld Guest from 54.147.189.54

Forum Moderators: goodroi

Message Too Old, No Replies

Googlebot Spoofing

How do you tell the real googlebot from spoofers?

     
6:03 pm on Dec 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 2, 2004
posts:71
votes: 0


My traffic reports show a lot of bandwidth going to various spiders claiming to be googlebot in their user-agent field. I'm pretty confident that the actual Googlebot wouldn't forget to close the parenthesis on the URL. Does anyone know definitively what Googlebot will call itself?

And is there any point in trying to ban the fake googlebots, or will those spiders just learn to spoof better?

6:08 pm on Dec 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 22, 2004
posts:1082
votes: 0


its like

Agent: Googlebot/2.1 (+http://www.google.com/bot.html)

Yahoo! Slurp; [help.yahoo.com...]

Agent: msnbot/0.3 (+http://search.msn.com/msnbot.htm)

6:29 pm on Dec 20, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 14, 2003
posts:4249
votes: 16


Maybe you could check all the server vars. Like if you see a gbot using a windows os or mac. This can be done in htaccess or php.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members