Googlebot and vbulletin

Goggle now spidering vbulletin?

[Splatt]

I had been wondering why our server load had been so high for the past hour with constant loads of 2+ as this is definetly our slowest time of day. Tried apache restart, tried rebooting the server, tried deleting all the apache log files.. still the same. Then I tried tail -f on one of the log files :

64.68.86.54 - - [15/Dec/2002:04:00:42 +0000] "GET /forums/search.php?s=SessionIdRemoved&action=finduser&userid=10395 HTTP/1.0" 200 2220 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [15/Dec/2002:04:00:42 +0000] "GET /forums/member2.php?s=SessionIdRemoved&action=addlist&userlist=ignore&userid=7532 HTTP/1.0" 200 10416 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [15/Dec/2002:04:00:42 +0000] "GET /forums/search.php?s=SessionIdRemoved&action=showresults&searchid=49135 HTTP/1.0" 200 24975 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.123 - - [15/Dec/2002:04:00:42 +0000] "GET /forums/member2.php?s=SessionIdRemoved&action=addlist&userlist=buddy&userid=10418 HTTP/1.0" 200 10416 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.86.54 - - [15/Dec/2002:04:00:42 +0000] "GET /forums/showthread.php?s=SessionIdRemoved&postid=165953 HTTP/1.0" 200 81896 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Sample output shows 5 requests in 1 second and 3 in one second from the same ip! I think the average was at least 2 a second.

This is the first time I've even seen googlebot try and spider the vb forums, it's nice to see even though lots of the pages visited seem pretty useless. I hope the good pages make it in the index next month though i'm wondering if any of it will make it in due to the random session ids.

Googlebot hits last month (site total) :

215 0.01% Googlebot/2.1 +http://www.googlebot.com/bot.html)

Googlebot hits this month (site total) :

8 94401 3.46% Googlebot/2.1 (+http://www.googlebot.com/bot.html)!

[Brett_Tabke]

All pages with the session ids will eventually be dropped from the index. It will stumble on all the duplicate content it runs into and classify the site as an uncrawlable dynamic site.