Forum Moderators: mack

Message Too Old, No Replies

MSN bot hammering a site

         

Mokita

2:25 am on Feb 24, 2006 (gmt 0)

10+ Year Member



One of our sites has about 100 pages that bots are allowed to crawl. It is a product site, so the vast bulk of pages relate to individual products, and they rarely change once uploaded and linked.

But the MSN bot has gone crazy. So far this month it has crawled 705 pages and asked for robots.txt 131 times. That means it has taken every page 7 times in about 3 weeks. Whereas, in the whole of Nov last year it was 293 + 58.

Could there be a reason for this huge increase? At least it is obeying robots.txt, which is a relief.

RichTC

11:44 am on Feb 25, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The bots working for once?