| MSNBOT going crazy
It's appetite for dynamic links seems unsasionable |
Orange_XL
msg:1533585 | 10:24 pm on Jun 25, 2003 (gmt 0) |
Has anyone else notice that MSNBOT goes wild on dynamic pages such as forums? I'm using phpBB and MSNBOT has already hit it serveral thousend times, while it is only a small forum with few sections and topics. It just seems to think that every new session or other variable is a new webpage.
|
jeremy goodrich
msg:1533586 | 10:26 pm on Jun 25, 2003 (gmt 0) |
Many search engines have done that to my sites in the past - as well as some that run bots with fake referrers :) If it's causing a performance issue with your site, I would add the IP(s) in question to your .htaccess file to limit their bandwidth sucking. You can always let them index your site later, as they decide / and or announce they are releasing a publically searchable index.
|
|
|
