Forum Moderators: goodroi
Here is the format I have to allow Googlebot and Teoma, but will disallow all others agents from specific files. Can anyone verify that the following?
User-agent: Googlebot
Disallow:
User-agent: MSNbot
Disallow:
User-agent: *
Disallow: /directory1/file.html
Disallow: /directory2/file.html
Thanks in advance.