Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- MSN's many cloaked bots. Again.
Dijkgraaf - 1:27 am on Oct 27, 2010 (gmt 0)
You don't need to disallow all non-existent URL's, only the ones you are getting hits on and don't want to get hits on anymore.
Thread source:: http://www.webmasterworld.com/search_engine_spiders/4182830.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com