homepage Welcome to WebmasterWorld Guest from 54.166.105.24
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Can anyone post a complete robots.txt file here?
PowerUp




msg:3330505
 7:56 am on May 4, 2007 (gmt 0)


System: The following message was cut out of thread at: http://www.webmasterworld.com/robots_txt/3310622.htm [webmasterworld.com] by goodroi - 7:31 am on May 4, 2007 (utc -5)


Hi, sorry for posting this noob question. Can anyone post a complete robots.txt file here? I don't know how to start and end a robots.txt file.

 

goodroi




msg:3330644
 12:41 pm on May 4, 2007 (gmt 0)

Robots.txt is simply a text file. There is no special beginning or ending. You just need to specify the user agent and then list what you want that user agent to disallow.

Many sites don't need a robots.txt file since the webmaster wants all content to be indexed. If you need to use a robots.txt you very likely need it customized to your personal situation. This is because your website directory/page structure is different from others.

If you want to see examples of robots.txt files just go visit your favorite websites and add robots.txt to the URL. Well that is except for [webmasterworld.com...] which has been turned into a blog.

PowerUp




msg:3330828
 3:40 pm on May 4, 2007 (gmt 0)

Then does that mean if I want the robot to follow my sitemap, all I need to do is add the following line and nothing else... and save it as txt?

"Sitemap: http://www.example.com/sitemap.xml"

goodroi




msg:3332937
 2:29 pm on May 7, 2007 (gmt 0)

As per [sitemaps.org...]

Specifying the Sitemap location in your robots.txt file
You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line:
Sitemap: <sitemap_location>

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved