msn only reads the two same files every time

msn started spidering my site but seem stuck now?

[carrier]

Everything went nice, msn started spidering my site, slowly first but increasing in speed. Then during the spidering session, the server I'm on went down for a few hours and when it finally got online again msn did not care for a while.

Now it has been visiting me for the last weeks but it just fetch robots.txt and the last two pages it grabbed before the server went down.

By looking at the accesslog I can see it's the same two pages that it grabbed before the server went down and now it's visiting a few times a day and always look at robots.txt, then fetch the two pages, wait for a while, fetch them again and then go away, just to come back a few hours later and do the same thing.

For the last days it has grabbed the same files, and only those files 85 times.

Now, is this normal behaviour or do you think msnbot is stuck in a loop or something because the server went down? Should I just wait for it to start spidering again or do you think I should send an email to Microsoft?

[goodroi]

Welcome to WebmasterWorld carrier!

First make sure your robots.txt still validates ok. Then make sure your pages are nicely interlinked. Then get more deep inbound links for your site. The worst case scenario is that somehow you created a spider trap and the MSNbot is trying to avoid it. Good luck.

[Prolific]

I noticed the same thing, MSN found my site from one link and spidered that page over and over again for about 2 weeks, it finally broke loose and devoured the rest of my site. Give it time, it'll make the rounds.

[carrier]

Well robots.txt validates ok and google has spidered the site without any problems so I guess I'll just have to wait for msnbot to break free from the loop by itself. Thanx for the help and fast response. :)