Welcome to WebmasterWorld Guest from 23.20.245.192

Forum Moderators: mack

Message Too Old, No Replies

MSN/Live Search indexing everything.

even if it's disallowed in robots.txt

     
8:45 pm on Dec 19, 2006 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


I've just noticed today that MSN/Live is listing pages and directories that have been excluded in robots.txt ( even in .htaccess ). Yahoo and Google aren't having this problem naturally (and I know there's nothing wrong on my end).

The unwanted results show the domain name... www. whatever .com as the site/title, and then the full-path URL. ( in regular searches )

These are not 301's or any other modified URL's, just simply pages that we don't want indexed/displayed in the results. ( the sites in question have been around for 6-10 years, and have never had this problem until MSN/Live this month ).

Where's Tim? :(

10:41 pm on Dec 19, 2006 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Whoops... sorry Tim... wrong company.

Where's MSNDude?

10:31 pm on Jan 3, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Guess I must be the only one with this problem... :(

I can't imagine why MSN is the only one indexing URL's that shouldn't be... ( Google, Yahoo, Ask, etc... don't )... something's broke. ( what a surprise! ).

We're talking about 11 different sites, so it's likely to be more than just a fluke.

1:22 pm on Jan 4, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 12, 2003
posts: 751
votes: 50


It is also showing a snippet of the url's result in live.com?
1:29 pm on Jan 4, 2007 (gmt 0)

Senior Member

joined:Mar 8, 2002
posts:2897
votes: 0


Yes - I have excluded pages in the live index. It is very probable that we only excluded them AFTER they were indexed. How about you?

<added> My examples show no snippet. But I can get them by typing unique text into the search query.

[edited by: Receptional at 1:32 pm (utc) on Jan. 4, 2007]

3:00 am on Jan 5, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:June 9, 2005
posts:354
votes: 0


I've seen this happening for a while, even on pages that have always had noindex meta tags, and on other pages I tested using robots.txt to block.

In this thread [webmasterworld.com] from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

I wonder where msndude is to.. haven't seen him for quite a while now.

1:02 am on Jan 6, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Receptional,

The pages in question for me have been exlcuded for the past 6-10 years ( since their inception )... yet all of a sudden, MSN/Live decides to show them. ( URL only at least). I do have indexed pages that link to them, but that shouldn't matter. Google indexes them as well, but labels them supplemental, and doesn't include them in the normal search index. ( which is what would be expected )... yet MSN/Live has begun to show them as 'normal' pages... some even showing up in the #1 and 2 spot for searches.

I think/hope it's a glitch on their part as this hadn't happened with them in the past.

Here's to hoping the issue is fixed.

1:14 am on Jan 6, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


LunaC:

In this thread from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

After reading the thread you mentioned, it surely sounds like the same issue. What a shame!

Seems like the only other alternative is to disallow MSN completely... until they decide to get things right.