Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- MSNBot intentionally requesting bogus file names.


KenB - 1:45 pm on Jan 14, 2010 (gmt 0)


Microsoft (or any other crawler for that matter) can't possibly know whether a URL exists or not on a website before crawling it.

Big search engines follow links they find on the Net, and often those can actually be bogus (but search engines don't know it until they crawl them) - this is normal situation in any large scale crawling, there is a special error code 404 designed just for that.

No this isn't a normal situation and no msnbot did not find these URLs on the Internet. I'm 100% positive that msnbot is intentionally manufacturing bogus URLs to test how the server responds to them and whether or not 404 errors are properly issued. As TheMadScientist pointed out this would be a valuable thing for both the search engine and webmaster to know. HOWEVER, they don't need to be testing for 404 responses multiple times each day.

Heck the bogus URL format is predictable enough that any half decent spammer could create a regular expression to make sure proper 404 errors were provided for these bogus URLs while still creating SERP spam by feeding 200 codes and fake pages for everything else.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4048908.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com