Forum Moderators: open

Message Too Old, No Replies

Effective way to block all SE crawlers

         

skuba

6:40 pm on Mar 10, 2006 (gmt 0)

10+ Year Member Top Contributors Of The Month



Hi,
I have created a test site for one of my clients. I don't want that test site to get crawled or indexed on search engines, so I need to block ALL robots.

I know that robots.txt alone won't do it. Do you know a better way to do it?

Do you know the rules I could use on .htaccess to block all crawlers?

Thanks so much

caveman

7:19 pm on Mar 10, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Put it on a dev server, behind a firewall, and PW protect it. Never had a problem doing that.

skuba

7:46 pm on Mar 10, 2006 (gmt 0)

10+ Year Member Top Contributors Of The Month



I don't have access to all that. It's on the same server with the same type of protection, it's just a subdomain.

I need to protect it.

Thanks

encyclo

8:19 pm on Mar 10, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you have .htaccess, you almost certainly can use Apache basic authentification. Just password-protect the whole subdomain. It doesn't matter if you have a one-letter password that you hand out to everyone in the company - no SE bot will be able to view the site. You could even set the prompt text to "Type User and Pass to view site" and you'd be OK.

skuba

9:44 pm on Mar 10, 2006 (gmt 0)

10+ Year Member Top Contributors Of The Month



Could you explain me how to do that?
Thanks

encyclo

10:54 pm on Mar 10, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Everything explained here:

[httpd.apache.org...]

skuba

10:58 pm on Mar 10, 2006 (gmt 0)

10+ Year Member Top Contributors Of The Month



Done, thanks