Welcome to WebmasterWorld Guest from 54.163.39.19

Forum Moderators: mack

Message Too Old, No Replies

MSN/Live Search indexing everything.

even if it's disallowed in robots.txt

     
8:45 pm on Dec 19, 2006 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


I've just noticed today that MSN/Live is listing pages and directories that have been excluded in robots.txt ( even in .htaccess ). Yahoo and Google aren't having this problem naturally (and I know there's nothing wrong on my end).

The unwanted results show the domain name... www. whatever .com as the site/title, and then the full-path URL. ( in regular searches )

These are not 301's or any other modified URL's, just simply pages that we don't want indexed/displayed in the results. ( the sites in question have been around for 6-10 years, and have never had this problem until MSN/Live this month ).

Where's Tim? :(

10:41 pm on Dec 19, 2006 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Whoops... sorry Tim... wrong company.

Where's MSNDude?

10:31 pm on Jan 3, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Guess I must be the only one with this problem... :(

I can't imagine why MSN is the only one indexing URL's that shouldn't be... ( Google, Yahoo, Ask, etc... don't )... something's broke. ( what a surprise! ).

We're talking about 11 different sites, so it's likely to be more than just a fluke.

1:22 pm on Jan 4, 2007 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 12, 2003
posts: 741
votes: 33


It is also showing a snippet of the url's result in live.com?
1:29 pm on Jan 4, 2007 (gmt 0)

Senior Member

joined:Mar 8, 2002
posts:2897
votes: 0


Yes - I have excluded pages in the live index. It is very probable that we only excluded them AFTER they were indexed. How about you?

<added> My examples show no snippet. But I can get them by typing unique text into the search query.

[edited by: Receptional at 1:32 pm (utc) on Jan. 4, 2007]

3:00 am on Jan 5, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:June 9, 2005
posts:354
votes: 0


I've seen this happening for a while, even on pages that have always had noindex meta tags, and on other pages I tested using robots.txt to block.

In this thread [webmasterworld.com] from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

I wonder where msndude is to.. haven't seen him for quite a while now.

1:02 am on Jan 6, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


Receptional,

The pages in question for me have been exlcuded for the past 6-10 years ( since their inception )... yet all of a sudden, MSN/Live decides to show them. ( URL only at least). I do have indexed pages that link to them, but that shouldn't matter. Google indexes them as well, but labels them supplemental, and doesn't include them in the normal search index. ( which is what would be expected )... yet MSN/Live has begun to show them as 'normal' pages... some even showing up in the #1 and 2 spot for searches.

I think/hope it's a glitch on their part as this hadn't happened with them in the past.

Here's to hoping the issue is fixed.

1:14 am on Jan 6, 2007 (gmt 0)

Senior Member

joined:May 21, 2002
posts:762
votes: 0


LunaC:

In this thread from a while back sounded like msndude was saying it's by design to list a URL-only listing no matter what meta tags or if robots.txt is used.

After reading the thread you mentioned, it surely sounds like the same issue. What a shame!

Seems like the only other alternative is to disallow MSN completely... until they decide to get things right.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members