MSN/Live Search indexing everything.

even if it's disallowed in robots.txt

[bobothecat]

I've just noticed today that MSN/Live is listing pages and directories that have been excluded in robots.txt ( even in .htaccess ). Yahoo and Google aren't having this problem naturally (and I know there's nothing wrong on my end).

The unwanted results show the domain name... www. whatever .com as the site/title, and then the full-path URL. ( in regular searches )

These are not 301's or any other modified URL's, just simply pages that we don't want indexed/displayed in the results. ( the sites in question have been around for 6-10 years, and have never had this problem until MSN/Live this month ).

Where's Tim? :(

[bobothecat]

Whoops... sorry Tim... wrong company.

Where's MSNDude?

[bobothecat]

Guess I must be the only one with this problem... :(

I can't imagine why MSN is the only one indexing URL's that shouldn't be... ( Google, Yahoo, Ask, etc... don't )... something's broke. ( what a surprise! ).

We're talking about 11 different sites, so it's likely to be more than just a fluke.

[rustybrick]

It is also showing a snippet of the url's result in live.com?

[Receptional]

Yes - I have excluded pages in the live index. It is very probable that we only excluded them AFTER they were indexed. How about you?

<added> My examples show no snippet. But I can get them by typing unique text into the search query.

[edited by: Receptional at 1:32 pm (utc) on Jan. 4, 2007]

[LunaC]

I've seen this happening for a while, even on pages that have always had noindex meta tags, and on other pages I tested using robots.txt to block.

In this thread [webmasterworld.com] from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

I wonder where msndude is to.. haven't seen him for quite a while now.

[bobothecat]

Receptional,

The pages in question for me have been exlcuded for the past 6-10 years ( since their inception )... yet all of a sudden, MSN/Live decides to show them. ( URL only at least). I do have indexed pages that link to them, but that shouldn't matter. Google indexes them as well, but labels them supplemental, and doesn't include them in the normal search index. ( which is what would be expected )... yet MSN/Live has begun to show them as 'normal' pages... some even showing up in the #1 and 2 spot for searches.

I think/hope it's a glitch on their part as this hadn't happened with them in the past.

Here's to hoping the issue is fixed.

[bobothecat]

LunaC:

In this thread from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

After reading the thread you mentioned, it surely sounds like the same issue. What a shame!

Seems like the only other alternative is to disallow MSN completely... until they decide to get things right.