- Search Engines
-- Search Engine Spider and User Agent Identification
- 4:25 am on Feb 12, 2013
No request for pages with msnbot-media/1.1 UA, all requests are for robots.txt alone.
On my site, msnbot-media stopped asking for pages in late August. (I mention it in the current incarnation of At Home With The Robots.) It's now only images and robots.txt in rock-steady alternation.
At the transition point, it picked up
but robots.txt for about a week and a half :)
Brought to you by