Forum Moderators: goodroi
I am having difficulty creating a robots.txt file and was wondering if anyone could help. I have scoured the resources available at www.robotstxt.org and still don't understand how to format the file to get it to do the following:
I have 3 folders:
/private
/default
/special
1. I want the Altavista spider (scooter) to crawl everything except /special and the Inktomi spider (slurp) to crawl everything except /default.
2. I also want to exclude these 2 spiders from /private.
3. Lastly I want all other spiders to crawl everything except /special and /private
My problem is that I am not sure if the robots.txt file works cumulatively or not. If it does not, I believe the file should look like:
User-agent: scooter
Disallow: /private/
Disallow: /special/
User-agent: Slurp
Disallow: /private/
Disallow: /default/
User-agent: *
Disallow: /private/
Disallow: /special/
If the robots is processed cumulatively, it would look something like this:
User-agent: *
Disallow: /private/
Disallow: /special/
User-agent: scooter
Disallow: /special/
User-agent: Slurp
Disallow: /default/
Is anyone an expert on this? Which one should I use, if either of them are correct? I really want to get this right first time so I don't have to wait for the spiders to come round again...
TIA
