Forum Moderators: mack
But the MSN bot has gone crazy. So far this month it has crawled 705 pages and asked for robots.txt 131 times. That means it has taken every page 7 times in about 3 weeks. Whereas, in the whole of Nov last year it was 293 + 58.
Could there be a reason for this huge increase? At least it is obeying robots.txt, which is a relief.