Welcome to WebmasterWorld Guest from

Forum Moderators: mack

Message Too Old, No Replies

MSNBOT going crazy

It's appetite for dynamic links seems unsasionable

10:24 pm on Jun 25, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 24, 2002
votes: 0

Has anyone else notice that MSNBOT goes wild on dynamic pages such as forums? I'm using phpBB and MSNBOT has already hit it serveral thousend times, while it is only a small forum with few sections and topics. It just seems to think that every new session or other variable is a new webpage.
10:26 pm on June 25, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jeremy_goodrich is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 4, 2000
votes: 0

Many search engines have done that to my sites in the past - as well as some that run bots with fake referrers :)

If it's causing a performance issue with your site, I would add the IP(s) in question to your .htaccess file to limit their bandwidth sucking. You can always let them index your site later, as they decide / and or announce they are releasing a publically searchable index.