homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots.txt elementary question

 9:54 pm on Mar 23, 2004 (gmt 0)

This is elementary, but I need to double-check to make certain I don't make a major blunder.

I have [mydomain.com...]

To keep spiders out of only the 'world' directory, I merely use the following robots.txt

User-agent: *
Disallow: /world/

Is this correct? And can I save the robots.txt file in either PC format or Unix format? Or must it be in Unix format?



 12:53 am on Mar 24, 2004 (gmt 0)

The format looks fine.

As for Unix or Windows formatting...

I don't honestly think it really matters as much as it used to - big search engines learn to adapt to fit the data they want to index, and for something as important as robots.txt being as flexible as possible makes perfect business sense.

- Tony


 5:29 am on Mar 24, 2004 (gmt 0)

Thanks. I'm a little paranoid, but I've read a few posts where people accidentally instructed search engines not to spider their site. I hadn't done a robots.txt for a year, and just wanted to double-check.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved