Forum Moderators: goodroi
I have [mydomain.com...]
To keep spiders out of only the 'world' directory, I merely use the following robots.txt
User-agent: *
Disallow: /world/
Is this correct? And can I save the robots.txt file in either PC format or Unix format? Or must it be in Unix format?
As for Unix or Windows formatting...
I don't honestly think it really matters as much as it used to - big search engines learn to adapt to fit the data they want to index, and for something as important as robots.txt being as flexible as possible makes perfect business sense.
- Tony