protected directory and spiders

where is htaccess?

[silverbytes]

Simple question: I donīt want my site indexed or being found by ANY search engines.

If I use the option protect directory in VERTEX control panel (unix hosting) will that be enough? any user is asked for a password in order to get access to protected directory so I think protecting " / " would be enough...
but it is?

Someone talked about htaccess... I donīt see that file when I connect via ftp in any folder plus I donīt know what to edit If I find it...

I thought in writting a robots.txt as follows:

User-Agent: *
Disallow: /*

Is that syntax right? Should I upload to /public_html/
(thats where my index.html file lives) or in any other directory?
Thanks!

[BlueSky]

If you password protect the upper level directory, no bot can get in. They will get a 401 like everyone else and will eventually go away. If your cgi-bin and subdomains are located separately, you'll need to password protect them too otherwise anything in there is fair game.

Many FTP client software have an option to display hidden dot files. An .htaccess though will not be seen unless you already created one before.

robots.txt works for keeping out good bots and should be placed in your upper directory. If you password protect the upper directory, you won't need it. If you don't want to password protect everything, then you can use this:

User-agent: *
Disallow: /

However, bad bads won't follow it.

[silverbytes]

Seems like placing robots.txt is the easiest way.
should be in /public_html/ and that would be enough?
Is that completely safe? ALL bots will respect it?
And... does bots really found sites not submited to se?

[Mark_A]

no robots.txt file will not keep people or engines out.

Only those robots which respect that file.
It will not keep someone from adding your site to a directory or blog or manually submitting it to an SE, or a bot finding it ignoring the robots file and indexing it all thence the meta search engines will find it in their indexes etc etc

The username and password example of .htaccess is a better solution. Much better.

[claus]

silverbytes, your /public_html/ is password protected. This means that if spiders try to get your robots.txt file then they will be prompted for a password. They will never see the robots.txt file. In other words, there's no point in having it if you password protect your whole site right from the document root.

>> does bots really found sites not submited to se?

Yes they do. They follow links from other sites to your site.

>> option protect directory in VERTEX control panel

That's essentially the same as htaccess, you just have a control panel to do it for you.

>> I think protecting " / " would be enough... but it is?

Yes. But make sure your subdirectories are also protected. Normally they are when a higher-level directory is.

/claus