Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- New MSN bot?


Pfui - 5:48 am on Jun 12, 2009 (gmt 0)


msnbot-65-55-214-150.search.msn.com
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)

Robots.txt? Yes-BUT

Despite getting this robots.txt --

User-agent: *
Disallow: /

-- "adidxbot" went straight for a subdir content page. Whereupon it was 403'd because only "msnbot" bots are allowed (and only in certain dirs, and maybe not any for very much longer).


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3931327.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com