Forum Moderators: goodroi

Message Too Old, No Replies

Allowing some bots, disallowing all others

         

xbase234

9:13 pm on Oct 8, 2004 (gmt 0)

10+ Year Member



I need to verify the correct syntax for allowing major crawlers, and disallowing all others.

Here is the format I have to allow Googlebot and Teoma, but will disallow all others agents from specific files. Can anyone verify that the following?

User-agent: Googlebot
Disallow:

User-agent: MSNbot
Disallow:

User-agent: *
Disallow: /directory1/file.html
Disallow: /directory2/file.html

Thanks in advance.

jdMorgan

5:16 pm on Oct 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This controls MSNbot and not Teoma, but other than that, it's OK.

Jim