lucy24 - 8:42 am on Feb 11, 2013 (gmt 0)
Can you live without bing traffic? ;)
If a legitimate search engine gets anything other than a 200 or 404 when they ask for robots.txt, they're supposed to hold off on crawling the site-- the, ahem, rest of it-- until the robots.txt problem is sorted.
Might be an interesting question to ask bing themselves. Wouldn't they have more time to crawl if they didn't spend so much time reading robots.txt? It's not that riveting is it?
It's all the same robot from the same place, right? They're not showing up from 479 different bing/msn IPs in succession. (I was going to add: And they don't have 479 different user-agents. I guess they do if you count all those MSIE non-robots, but those don't ask for robots.txt in the first place.) They can hardly pretend they haven't seen it. Is there some long-standing bug in the software that's supposed to keep track of robots.txt pickups?
Hm. Wonder if they'd keep asking just as often if you started serving up mendacious 404s. (Not 410: bing doesn't treat those differently from 404s-- and in this case you wouldn't want them to!) Slight problem of course if there are areas of your site/s that you actually don't want bing/msn to crawl...