Welcome to WebmasterWorld Guest from 54.162.239.134

Forum Moderators: goodroi

Message Too Old, No Replies

Googlebot Spoofing

How do you tell the real googlebot from spoofers?

     
6:03 pm on Dec 20, 2004 (gmt 0)

10+ Year Member



My traffic reports show a lot of bandwidth going to various spiders claiming to be googlebot in their user-agent field. I'm pretty confident that the actual Googlebot wouldn't forget to close the parenthesis on the URL. Does anyone know definitively what Googlebot will call itself?

And is there any point in trying to ban the fake googlebots, or will those spiders just learn to spoof better?

6:08 pm on Dec 20, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



its like

Agent: Googlebot/2.1 (+http://www.google.com/bot.html)

Yahoo! Slurp; [help.yahoo.com...]

Agent: msnbot/0.3 (+http://search.msn.com/msnbot.htm)

6:29 pm on Dec 20, 2004 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Maybe you could check all the server vars. Like if you see a gbot using a windows os or mac. This can be done in htaccess or php.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month