homepage Welcome to WebmasterWorld Guest from 54.226.80.196
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Creating A Robots.Txt File
How Do I Create A Correct robots.txt File?
Jackal




msg:1528066
 6:33 am on Feb 14, 2005 (gmt 0)

Hey Everyone,

I am new to this site. Just wanted to say hello and that this is an excellent site! I also have several questions and was hoping to get some feedback on them from some of the experienced webmasters in here.

1. How do I create a good correct robots.txt file? Which websites or discussions would assist me learn as much as possible about this broad topic and others associated with it?

2. What information resources (i.e. websites and books) can I use for reference for becoming a better webmaster?

3. Which websites are the best for researching information on SEO? I have purchased a program from my hosting provider for this called Traffic Blazer and wanted to know what other things I can do other than this since using more tools would help elevate the rankings I am interested in.

4. Which websites are the best for cross-referencing "visitors" and their IP Addresses to identify those who are doing all of the nefarious things many here say they are doing when visiting a website one has designed?

Thanks for your help in advance! :)

Jackal

 

rdmedia




msg:1528067
 3:48 pm on Feb 14, 2005 (gmt 0)

this looks like a nice place to put my first post :)

Also a SEO n00b so this is one of my burning questions too.

jetboy




msg:1528068
 4:27 pm on Feb 14, 2005 (gmt 0)

Welcome to WebmasterWorld folks! (oh look, me too ;) )

As this is a robots.txt forum, you might get more joy in some of the other forums for your other questions. Briefly:

1. A robots.txt file is just a simple text file you can create in Notepad (called robots.txt of course) uploaded to the root of you site. The majority of search engine spiders request this file to see where they are allowed on the site. For the syntax to use, either look through the archives on this forums, or check out The Web Robots Pages (at [robotstxt.org ].)

What you put in it depends on what you are trying to achieve. A couple of basic ones are:

To exclude all robots from the entire server:

User-agent: *
Disallow: /

(i.e. '*' is a wildcard match for all robots, and '/' means the roots directory, and consequently any subdirectories of the site.)

To allow all robots complete access

User-agent: *
Disallow:

(i.e. Again, '*' is a wildcard match for all robots, and no directories are specified. If you just want a robots.txt file for the sake of having one, this is what to put in it.)

2. There have been a number of threads on this topic in the past. Your best bet is to search for something like 'best books' using the WebmasterWorld site search (link at the top of the page). For the record, I think every webmaster should read Steve Kruq's Dont Make Me Think and 37 Signals' Defensive Design For The Web.

3. For the newbie, you're probably at the best place you're going to find already. While I'm not going to knock Traffic Blazer, as I've never used it, the advice you'll get here is to be very careful if you use any sort of automated optimization or submission tools. There are some things you should be doing manually (optimization) and there are some things that aren't going to do you any good at all (repeated SE submission).

4. Nefarious things? Such as? Don't worry about it. Really. You're only likely to become a target if you're a player in a competitive industry. If you do become a target, checking IP addresses is unlikely to do you any good. ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved