Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- msnbot-media

lucy24 - 1:42 am on Dec 24, 2012 (gmt 0)


Well, no wonder msnbot-media isn't robots.txt compliant. You won't let it see robots.txt in the first place :)

<Files "robots.txt">
Order Allow,Deny
Allow from all
</Files>

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4470273.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com