Forum Moderators: goodroi

Message Too Old, No Replies

Allowing and disallowing spiders

Easy one - can somebody confirm my understanding?

         

Zeberdy

3:20 pm on Apr 1, 2005 (gmt 0)

10+ Year Member



Just a quickie…..
Does the following allow all robots to visit all files?

User-agent: *
Disallow:

Whilst this one keeps all robots out.

User-agent: *
Disallow: /

Easy one for you guys - I know - but I need clarification for the sake of my sanity....

I am trying to stop spiders from enter test sites.

Span

5:42 pm on Apr 1, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



User-agent: *
Disallow: /

..will only stop spiders that read and obey your robots.txt. So for the sake of your sanity it wouldn't hurt to password protect your test sites.

Zeberdy

9:19 am on Apr 21, 2005 (gmt 0)

10+ Year Member



Yeah good idea - thanks for the reply.

Sorry for the delay in my response.