Welcome to WebmasterWorld Guest from 50.19.156.19

Forum Moderators: goodroi

Message Too Old, No Replies

Can anyone post a complete robots.txt file here?

     

PowerUp

7:56 am on May 4, 2007 (gmt 0)

5+ Year Member




System: The following message was cut out of thread at: http://www.webmasterworld.com/robots_txt/3310622.htm [webmasterworld.com] by goodroi - 7:31 am on May 4, 2007 (utc -5)


Hi, sorry for posting this noob question. Can anyone post a complete robots.txt file here? I don't know how to start and end a robots.txt file.

goodroi

12:41 pm on May 4, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Robots.txt is simply a text file. There is no special beginning or ending. You just need to specify the user agent and then list what you want that user agent to disallow.

Many sites don't need a robots.txt file since the webmaster wants all content to be indexed. If you need to use a robots.txt you very likely need it customized to your personal situation. This is because your website directory/page structure is different from others.

If you want to see examples of robots.txt files just go visit your favorite websites and add robots.txt to the URL. Well that is except for [webmasterworld.com...] which has been turned into a blog.

PowerUp

3:40 pm on May 4, 2007 (gmt 0)

5+ Year Member



Then does that mean if I want the robot to follow my sitemap, all I need to do is add the following line and nothing else... and save it as txt?

"Sitemap: http://www.example.com/sitemap.xml"

goodroi

2:29 pm on May 7, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



As per [sitemaps.org...]

Specifying the Sitemap location in your robots.txt file
You can specify the location of the Sitemap using a robots.txt file. To do this, simply add the following line:
Sitemap: <sitemap_location>