homepage Welcome to WebmasterWorld Guest from 54.204.58.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Fast not checking robots META tag
SeanL




msg:224323
 3:57 pm on Apr 15, 2003 (gmt 0)

Hello, all. This is my first time posting to this forum, though I have learned so much by reading it.

I have one page of links I do not want followed. I put in the following tag in the head:

<META name="robots" content="noindex,nofollow">

Unfortunately, Google and Fast seem not to obey it. I submited a bug report to Fast and the reply just came back that they would have their bot check my robots.txt again. They did not mention the issue I raised, which was the robots meta tag. I have altered my robots.txt, but the meta tag is supposed to work by itself, isn't it?

So, I figure this might be of interest to some here. I searched through the Fast forum to see if this issue had been mentioned, but I didn't see anything. Has anyone else noticed this behavior, or does their bot seem to behave appropriately on your site?

And also, does anyone see a flaw in the tag above?

-- SeanL

 

Rumbas




msg:224324
 4:18 pm on Apr 15, 2003 (gmt 0)

Hello Sean, welcome to the board.

I've not heard or seen that Fast should not honor the exclusion tags. Neither robots.txt nor meta name noindex.

At least that's what their FAQ says [fastsearch.com].

However they list the code for the meta tag with spaces between noindex and nofollow:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Could be the reason they still index your page?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved