Forum Moderators: mack

Message Too Old, No Replies

MSNBot and Folder Errors

         

rogerd

9:19 pm on Oct 7, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



On one site I manage, I noticed a series of weird requests while checking 404 errors. They were of the form,

http://www.example.com/apple/apple/apple/apple/orange.htm

There is, in fact, a page like this:
http://www.example.com/apple/orange.htm

At first I thought it was some kind of misguided hack attempt, but the raw logs showed it was only the friendly MSNBot. This bot (no problem with Googlebot or Slurp) seems to have gone a bit nutty, asking for various numbers of non-existent folders. I.e., it looked for the same page with two /apple/ folders, three, four, etc. up to six or so.

Anyone know what can trigger this kind of behavior? The links are coded as relative, but I don't see any way that the bot could get fed recursively deeper false directories. Or should I just chalk it up to a random bot error and not worry about it if it doesn't happen again?

Receptional

7:46 am on Oct 8, 2005 (gmt 0)



That might account for MSNBot being such an aggressive little number. I can easily imagine the msnbot program doing an iterative index of a page and continually adding a subdirectory in the tree every time it returns to link 2 on the page, link 3 on the page etc.