Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- msnbot-media


wilderness - 3:09 am on Dec 24, 2012 (gmt 0)


Hey lucy,
I'm not counting the msnbot-media requests for a day again (see earlier reply this thread).

The only msnbot-media robots.txt request that fail are the failed headers checks, which only come from these 3x Class three ranges. All the others get through.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4470273.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com