- Search Engines
-- Search Engine Spider and User Agent Identification
---- msnbot-media/1.1 generating a ton of 404s
- 1:35 am on Dec 11, 2013
I am going the soft road here first via robots.txt
take it from an experiences "softie", and don't waste your time with robots.txt and the msn-media bot.
Brought to you by