Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- msnbot-media


keyplyr - 8:41 pm on Feb 11, 2013 (gmt 0)


Isn't this one of the things search engines do on purpose to make sure you haven't got any lurking Soft 404s?

For that purpose, I have seen things like:

http://example.com/NO-EXIST.html

What I'm referring to is this:

http://example.com/random-numbers-here/valid-page.html

Bing says it's not them, that they may be following links from some directory who may have a toxic data-base and that these bad links will not affect my indexing and will eventually drop-off their crawl. They have not.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4470273.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com