How to Disallow a Particular Page

Forum Moderators: goodroi

Message Too Old, No Replies

How to Disallow a Particular Page

jsnively

7:05 pm on Oct 6, 2004 (gmt 0)

I searched this forum and can't find the answer to a really simple question: How do I keep one particular page on my site from being spidered?

moltar

7:11 pm on Oct 6, 2004 (gmt 0)

[url=http://www.robotstxt.org/wc/faq.html#nosecurity]Disallow: /path/to/page.html[/url]

jsnively

8:33 pm on Oct 6, 2004 (gmt 0)

Thank you, moltar. Is there a way to do this on the page I don't want spidered? If so, what is the code and where does it go?

Larryhat

9:16 pm on Oct 6, 2004 (gmt 0)

Inside the <head> at the top of an HTML page,
near the beginning of head, insert the line:

</head>

I use it, been told it works, but I cannot prove it
because I never send that test page up to my host ISP.
I just have it in case I accidentally upload the page.

- Larry

moltar

9:25 pm on Oct 6, 2004 (gmt 0)

Yes, Larryhat, that works with major SEs. Also major SEs have specific meta tags. For example, if you do not want only Google to spider your page, you can say:

<meta name="googlebot" content="noindex" />