How to make a page *inaccessible* to searches

without using password protection?

[halfandhalf]

Hi,
I need to create a page... the URL will be sent to a select few, so we don't want people to come across it by searching. Is this even possible or does the page have to be password protected?

Thanks.
h+h

[Nick_W]

Sure, just add a robots.txt [webmasterworld.com] file to your http document root (where you keep your pages).

The file should look like this:

User-agent: *
Dissallow /

That'll stop all bots.

Nick

[bcolflesh]

That'll stop all bots.

That'll stop all bots who obey/read the robots.txt file - don't count on this method - don't put data in a public place that you don't want to be "found".

[halfandhalf]

wow, thanks for that. can i do that in a separate directory, so that only the page in that directory is inaccessible to the bots?

[Nick_W]

Disallow /dir/page.html

I *think* that'd do it...

Nick

[halfandhalf]

And does the robots.txt page go within that directory or still at the root?

[heini]

I'd say password protections is by far the better method in this case. One of the reasons rogue bots are called rogue is that they don't obey to robots.txt.

[jomaxx]

Don't use Nick_W's first suggestion unless you want to ban ALL good bots from your site, which I don't think is your intention. His second suggestion would also work, but has the downside of telling the whole world the name of this secret page.

I suggest protecting a subdirectory using the sample below, and then putting the page there and giving it a more-or-less unguessable name (i.e. not index.html). Also make sure that if someone nosy tries to access that directory, that your server doesn't cough up a list of the files in the directory. That should be sufficient unless the data is really sensitive, in which case you should definitely password-protect it.

User-agent: *
Dissallow /some_private_dir/

[halfandhalf]

thanks for your help!

[jomaxx]

**** TYPO ALERT: "Disallow" has just one "s" in it. I cut-and-pasted part of that code from an earlier post without proofreading it. Use this instead:

User-agent: *
Disallow: /some_private_dir/

[Mohamed_E]

If all you want to do is make sure the page is not advertized to the whole world in the major search engines then I suppose that a suitable robots.txt will suffice.

If you want to be sure that nobody unauthorized can read it there is simply no alternative to a password protection scheme.

[rasslin russ]

One trick I've heard is to link to the new url using a button.onClick command or a nested form with a submit button to go to the page in question. Most robots (and please correct me if I'm wrong) don't follow these types of links.