Forum Moderators: goodroi

Message Too Old, No Replies

Quick robots.txt help

is this the right syntax?

         

Fuzzi_Bear

4:41 pm on Nov 23, 2005 (gmt 0)

10+ Year Member



Hi,

I want to disalow robots from 'images' and 'cgi-bin' but allow access to everything else.

is this the correct syntax?

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow:

thanks.

DanA

5:48 pm on Nov 23, 2005 (gmt 0)

10+ Year Member



Your last line is not necessary.

Fuzzi_Bear

2:17 pm on Nov 25, 2005 (gmt 0)

10+ Year Member



I see, So 'unless' there is a disalow entry for something, robots assume they can spider it?

Thanks