why msnbot 1.0 always spider currently no exist directory

msnbot 1.0

[johnlim9988]

Hi,

why msnbot 1.0 always spider currently no exist directory?

Beofore we have a links directory then we delete that directory about half an year ago. But MSNbot always spider that directory until now.

Such behavior of MSNbot have any side effect? Is this the reason MSN not cache our sites? Should I ban the MSNbot to spider that no exist directoy at robots.txt?

[msndude]

There are probably a lot of other pages that still link back to the non-existent directory. MSNbot sees those links and follows them. This doesn't really hurt anything, but it does waste a bit of time. If you block it in your robots.txt file that would probably benefit both you and us, but it should have zero impact on your ranking.

Hope that helps!