Welcome to WebmasterWorld Guest from 50.19.156.19

Forum Moderators: mack

Message Too Old, No Replies

MSNBOT going crazy

It's appetite for dynamic links seems unsasionable

     

Orange_XL

10:24 pm on Jun 25, 2003 (gmt 0)

10+ Year Member



Has anyone else notice that MSNBOT goes wild on dynamic pages such as forums? I'm using phpBB and MSNBOT has already hit it serveral thousend times, while it is only a small forum with few sections and topics. It just seems to think that every new session or other variable is a new webpage.

jeremy goodrich

10:26 pm on Jun 25, 2003 (gmt 0)

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Many search engines have done that to my sites in the past - as well as some that run bots with fake referrers :)

If it's causing a performance issue with your site, I would add the IP(s) in question to your .htaccess file to limit their bandwidth sucking. You can always let them index your site later, as they decide / and or announce they are releasing a publically searchable index.