Forum Moderators: open

Message Too Old, No Replies

Hide page from spiders

... for an html newby

         

JonnyWales

2:43 pm on Jun 9, 2003 (gmt 0)

10+ Year Member



I've looked in Search but cannot find exactly what I'm looking for.

All I want to do is have 2-3 pages which will not have links from the main part of the site, ie. visitors will only reach them by clicking on a special offer link in an email message. So I don't want these 2-3 pages to get spidered and subsequently appear in an index.

How do I do this?

korkus2000

2:45 pm on Jun 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member


You should put these files in a directory and make it robots excluded in your robots.txt.

http://www.robotstxt.org/wc/robots.html

JonnyWales

5:44 pm on Jun 9, 2003 (gmt 0)

10+ Year Member



Thanks for that. One small supplementary :-

If I wish to exclude the spiders from seeing a page which also resides in the root along with index.htm which is the correct syntax?

User-agent: *
Disallow: page_to_hide.htm

OR

User-agent: *
Disallow: /page_to_hide.htm

g1smd

10:07 pm on Jun 9, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If I don't want spiders to index one or two particular HTML pages, then I add this meta tag only to those pages:

<meta name="robots" content="noindex,follow">

You might want nofollow in place of follow if you also do not want the spiders to follow any links on those pages.