Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt elementary question



9:54 pm on Mar 23, 2004 (gmt 0)

10+ Year Member

This is elementary, but I need to double-check to make certain I don't make a major blunder.

I have [mydomain.com...]

To keep spiders out of only the 'world' directory, I merely use the following robots.txt

User-agent: *
Disallow: /world/

Is this correct? And can I save the robots.txt file in either PC format or Unix format? Or must it be in Unix format?


12:53 am on Mar 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

The format looks fine.

As for Unix or Windows formatting...

I don't honestly think it really matters as much as it used to - big search engines learn to adapt to fit the data they want to index, and for something as important as robots.txt being as flexible as possible makes perfect business sense.

- Tony


5:29 am on Mar 24, 2004 (gmt 0)

10+ Year Member

Thanks. I'm a little paranoid, but I've read a few posts where people accidentally instructed search engines not to spider their site. I hadn't done a robots.txt for a year, and just wanted to double-check.

Featured Threads

Hot Threads This Week

Hot Threads This Month