Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
I have one page of links I do not want followed. I put in the following tag in the head:
<META name="robots" content="noindex,nofollow">
Unfortunately, Google and Fast seem not to obey it. I submited a bug report to Fast and the reply just came back that they would have their bot check my robots.txt again. They did not mention the issue I raised, which was the robots meta tag. I have altered my robots.txt, but the meta tag is supposed to work by itself, isn't it?
So, I figure this might be of interest to some here. I searched through the Fast forum to see if this issue had been mentioned, but I didn't see anything. Has anyone else noticed this behavior, or does their bot seem to behave appropriately on your site?
And also, does anyone see a flaw in the tag above?
joined:Oct 23, 2000
I've not heard or seen that Fast should not honor the exclusion tags. Neither robots.txt nor meta name noindex.
At least that's what their FAQ says [fastsearch.com].
However they list the code for the meta tag with spaces between noindex and nofollow:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Could be the reason they still index your page?