-- Search Engine Spider and User Agent Identification
---- msnbot-media/1.1 generating a ton of 404s
lucy24 - 8:25 pm on Jan 2, 2014 (gmt 0)
The only thing in robots.txt that you can absolutely demand everyone follow is "Disallow". Everything else is an optional extra. I don't know about bing, but the googlebot says outright that it disregards "Crawl-Delay". You have to set it in wmt instead.