Forum Moderators: phranque

Message Too Old, No Replies

.htaccess and robots.txt?

         

duden

2:46 pm on Apr 17, 2006 (gmt 0)

10+ Year Member



Hi,

I'm trying to disallow robots.txt from viewing with .htaccess like that:

<Files .htaccess>
order allow,deny
deny from all
</Files>

<Files robots.txt>
order allow,deny
deny from all
</Files>

Now, my question is : could the robots read the robots.txt and if not does it mean that they will index the whole site?

Thanks.

jdMorgan

5:43 pm on Apr 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes, if they cannot read robots.txt for any reason, then that means that they should fetch all URLs on your site.

Jim

duden

6:25 pm on Apr 17, 2006 (gmt 0)

10+ Year Member



OK, Thanks!