Hi, I have created a test site for one of my clients. I don't want that test site to get crawled or indexed on search engines, so I need to block ALL robots.
I know that robots.txt alone won't do it. Do you know a better way to do it?
Do you know the rules I could use on .htaccess to block all crawlers?
Thanks so much
caveman
7:19 pm on Mar 10, 2006 (gmt 0)
Put it on a dev server, behind a firewall, and PW protect it. Never had a problem doing that.
skuba
7:46 pm on Mar 10, 2006 (gmt 0)
I don't have access to all that. It's on the same server with the same type of protection, it's just a subdomain.
I need to protect it.
Thanks
encyclo
8:19 pm on Mar 10, 2006 (gmt 0)
If you have .htaccess, you almost certainly can use Apache basic authentification. Just password-protect the whole subdomain. It doesn't matter if you have a one-letter password that you hand out to everyone in the company - no SE bot will be able to view the site. You could even set the prompt text to "Type User and Pass to view site" and you'd be OK.