Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- Bing ignoring robots.txt?


lucy24 - 10:24 pm on Aug 5, 2012 (gmt 0)


Your post is kinda fuzzy on the difference between crawling and indexing. Entirely different processes. I do know from personal observation that bing doesn't seem to care about the "noindex" instruction in a meta tag. But it has to crawl the page in order to read-- or ignore, as the case may be-- anything on it including the meta.

Last time I looked, yahoo was using bing's information rather than do their own crawling.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4441555.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com