Forum Moderators: phranque

Message Too Old, No Replies

Is there a way of keeping ALL search engines spiders out

Without using a logon?

         

Smiley

9:59 am on Feb 25, 2003 (gmt 0)

10+ Year Member



I want to hide a whole website from the search engines without using a logon so users will only be able to use this site if they know the URL.

Is there a 100% sure way of doing this?

Thanks!
Smiley

hakre

10:55 am on Feb 25, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



hi Smiley,

you can combine multiple methods to gain it 99%

- disallow per robots.txt
- disallow per meta headers
- block ip's of known robots
- block known spider via the useragents info

the ip-blocking might be the most powerfull feature because you're shure the spiders can't connect to your site. but your list has to be up-to-date for this.

there are some comprehensive and mostly completely lists of such ip-adresses and .htaccess files for this available in this forum. please utilize the site search to locate them.

the Search Engine Spider Identification Forum [webmasterworld.com] is a good choice on this topic.