| Googlebot Spoofing
How do you tell the real googlebot from spoofers? |
teaperson
msg:1526234 | 6:03 pm on Dec 20, 2004 (gmt 0) |
My traffic reports show a lot of bandwidth going to various spiders claiming to be googlebot in their user-agent field. I'm pretty confident that the actual Googlebot wouldn't forget to close the parenthesis on the URL. Does anyone know definitively what Googlebot will call itself? And is there any point in trying to ban the fake googlebots, or will those spiders just learn to spoof better?
|
benevolent001
msg:1526235 | 6:08 pm on Dec 20, 2004 (gmt 0) |
its like Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
Yahoo! Slurp; [help.yahoo.com...] Agent: msnbot/0.3 (+http://search.msn.com/msnbot.htm)
|
ogletree
msg:1526236 | 6:29 pm on Dec 20, 2004 (gmt 0) |
Maybe you could check all the server vars. Like if you see a gbot using a windows os or mac. This can be done in htaccess or php.
|
|
|
