Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- MSNBot posing as Internet Explorer 7 and not reading robots.txt?
robzilla - 4:49 pm on Jun 25, 2013 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/4587076.htm
|you might recheck your logs and see if these are using the broken UA with the double-trialing spaces. |
They are, indeed. Thanks for that, makes it easier to block these requests from my statistics.
Brought to you by WebmasterWorld: http://www.webmasterworld.com