Welcome to WebmasterWorld Guest from 54.221.49.52

Forum Moderators: mack

Message Too Old, No Replies

Does msnbot obey meta tag noindex?

     

morags

9:46 am on Oct 17, 2006 (gmt 0)

10+ Year Member



Further to the thread below - but slightly different:

How do you prevent msnbot from indexing a page? This page has "<meta name="robots" content="noindex,nofollow">" in the head. ALL bots are also excluded from this page via robots.txt. And yet the page IS indexed and appears on search.live.com.

Google and Yahoo ignore this page as expected. How do I prevent msnbot from indexing the page? Do I need to specifically mention msnbot in robots.txt?

I am becoming very worried about possible duplicate content problems.

Thanks.

msndude

4:01 am on Oct 18, 2006 (gmt 0)

10+ Year Member



May I ask what sort of "duplicate content problem" you are worrying about? In general, we view things like robots.txt as a request that we leave you alone -- if you ban us in your robots.txt, then we won't crawl your site. Period. If that is not the case, then it's a serious bug, and you should let me know immediately.

However, if enough sites link to you, we'll index your site anyway -- without crawling you -- just because you must be interesting to someone. You need to have a LOT of links from a lot of different non-spam people for that to happen, though. It's kind of like being a celebrity; if you're famous, you can't really complain that people are talking about you.

LunaC

2:14 pm on Oct 18, 2006 (gmt 0)

10+ Year Member



msndude, I disagree very strongly with MSN's stance on this.

If I say noindex, nofollow.. I DON'T want that page listed, not as a URL, not in the SERP in any way. I don't care if people are talking about it or not.

And as for "You need to have a LOT of links from a lot of different non-spam people for that to happen, though.", again, I disagree. I have pages that are only linked from my own site, that clearly are disallowed (meta noindex, nofollow.. and always have been) that are listed.

One example is a contact form. I don't want people coming in on that page since there's no good reason to contact if they haven't been to the rest of the site. The only reason people look for contact forms is to spam, find potential vulnerabilities and a myriad of other unpleasant things... I see no good reason for this to be an entry page.

This page is not linked to from any outside source. It has always had noindex,nofollow... so why is it even listed?

This isn't the only example I've found either, I have many pages that are only of use if the visitor has been *in* the site.. coming in on them is useless.. so I disallow... still, they are listed in the search listings. (Yes, URL only, but that is clearly indexed!) I also worry about duplicate content on these.

An example is a stock photo archive. The pages that are disallowed are the full image, allowed are the pages with the thumbnails and a brief, but useful description of each image. Since the full image pages would be very thin on spiderable content (simply a large image and very minimal navigation), that would be useless as a landing page, and very much open to duplicate content. Therefore.. noindex, nofollow would be added. So why would they be listed?

Still using a stock photo archive as an example, say it has 100 pages of thumbnails, each listing 10 images. MSN would list 1000 urls, 100 descriptive listings equaling a total (ignoring fluff pages such as FAQ etc.) of 1100 listings. Now I have 1100 pages to try to sift through to find the item I was looking for instead of 100.

I know from experience that if I use site: to find pages listed, the full listings do not rise to the top, not even the pages I'm more than likely looking for.. ie, the most commonly viewed... how is that useful? (Don't believe me? try site:www.microsoft.com, how is that helpful?)

What use is it to searchers to get this information? Since there must be a real reason that MSN has decided to list disallowed pages and URL only listings, I'd like to hear why MSN has decided listing this content is to the benefit of anyone, either searchers, website owners.. or ANY good reason.

As for being a celebrity and not being able to complain about people talking.. ok.. to a degree I can agree with that. I can *talk* all I want about a celebrity... but peeking into their windows, hinting at what I see and giving their private address IS NOT acceptable.

jimbeetle

3:16 pm on Oct 18, 2006 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



This page has "<meta name="robots" content="noindex,nofollow">" in the head. ALL bots are also excluded from this page via robots.txt. And yet the page IS indexed and appears on search.live.com.

Pages that are excluded by robots.txt can appear as URL-only results in the SERPs.

Pages that carry the "noindex" robots meta should not appear in the SERPs as URL only or anything else.

But, since the robots.txt directive is the first thing that a bot sees it does not retrieve the page, so does not see the "noindex" robots meta.

You have to remove the block in the robots.txt so the bot can see the "noindex".

Note that Googlebot works the same way, not sure about Y!.

LunaC

8:12 pm on Oct 18, 2006 (gmt 0)

10+ Year Member




That's what I'd thought to, but I don't have any bots blocked by robots.txt. I block with meta tags only, yet those are definitely appearing as urls in MSN's search, and not showing at all in Y. or G. (which is what I'd want, no listings for those pages in any way).

jimbeetle

8:20 pm on Oct 18, 2006 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



This page has "<meta name="robots" content="noindex,nofollow">" in the head. ALL bots are also excluded from this page via robots.txt. And yet the page IS indexed and appears on search.live.com.

Just going by what you stated above.

LunaC

8:58 pm on Oct 18, 2006 (gmt 0)

10+ Year Member



Oh, no... I think some wires got crossed in this thread. That was morags that wrote that (robots.txt and meta both used on his/her site).

I missed that when I wrote my post, however my sites don't have robots.txt blocking anything. I'm just using meta tags (noindex, nofollow) and those pages are being indexed (url only) in MSN.

My post was to find out why MSN ignores noindex, as msndude said "if enough sites link to you, we'll index your site anyway -- without crawling you -- just because you must be interesting to someone." and if there is any way to stop it from happening. Perhaps "noindex!important, nofollow" could be implemented if we really mean, "don't index this page" ;)

[edited by: LunaC at 9:00 pm (utc) on Oct. 18, 2006]

jimbeetle

9:15 pm on Oct 18, 2006 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



wires crossed

Yeah, mine. I thought you were the OP replying to my reply to the OP ;-).

I'm assuming that, in context, when msndude said, "if enough sites link to you, we'll index your site anyway -- [b]without crawling you[b]," he was referring to to a URL-only listing when the bot is excluded by robots.txt.

If your meta robots elements are correctly formed, and if you still see URL-only listings for those pages, it's probably time to drop a dime and send msndude some concrete examples.

jdMorgan

9:28 pm on Oct 18, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Most other major SEs take the <meta name="robots" content="noindex"> tag to mean "Don't index this URL" that is, "Don't list or mention this URL in any way." However, MSN/Windows Live treats it just like a robots.txt Disallow, and will index a URL-only link.

The only way to stop them is by cloaking the page for their robot, redirecting it to some other URL that has a real listing. :(

Now that they've apparently got their robots.txt compliance sorted, perhaps they can address this problem.

However, beware of a common problem before blaming MSN/Live: In order to obey an on-page meta-robots tag, the robot must be allowed to fetch that page. So do not disallow a page's URL in robots.txt if you want SEs to comply with meta-robots tags on that page.

Jim

jimbeetle

1:39 pm on Oct 19, 2006 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



However, MSN/Windows Live treats it just like a robots.txt Disallow, and will index a URL-only link

Whoa, I haven't seen this behaviour as yet. Seems like MSN is taking all control out of webmasters' hands. Might be time for certain people in the industry to get together again to discuss compliance with standards, or what are supposed to be standards.

LunaC

3:45 pm on Oct 19, 2006 (gmt 0)

10+ Year Member




What upsets me is that from what msndude said in his post above, it sounds like that's behavior "by design", not just a bug.

morags

1:18 am on Oct 22, 2006 (gmt 0)

10+ Year Member



An update:

All now looks as it should. I really did see a "noindex" page in the SERPS - not just URL-only. But all have disappeared.

MSNdude - the "dup content worry" was because I have 5 different ways of displaying the same page/content. 4 have "noindex" set - so SEs only get the one standard page (users can reformat if they like after they have arrived). But last week I noticed one ot the "noindex" pages in search.msn.com - hence the panic.

And thanks for putting me straight on robots.txt/noindex meta conflict. Will let bots in to see the noindex.

Thanks all.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month