homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Beginners guide to robots
Help needed!

 10:43 am on Sep 29, 2002 (gmt 0)

Dear webmasters,

As a complete amateur, our web site may look a bit shabby, but right now we can't afford fancy web design. However, largely thanks to this Wemaster site, we get high rankings for our keywords, and I am still trying to optimise. The site has only been online since June, and I am gathering links and hopefully customers!

Anyway, I am very confused about all this robots.txt talk, and want to know if we have one or not! NO and we don't need one!I think it's the bit which says :

<meta name="robots" content="index,follow">

Yes? NO! That's the robots meta tag. Wow, you learn more every day!

If I want to disallow robots, where does this code go? Under the previous line? Is it worth disallowing robots?

Any pros got any comments on our site? Am I doing anything really wrong? Our market is very competitive on the ground, but most of the competing websites are not so great, and a PR of 5 is about as high as you'll find in our field (we are currently PR 4).

Sorry to bother y'all like this.

Have a nice day!

Hey! I can't see my homepage on my profile! How can anyone help me!

[edited by: Monkscuba at 1:26 pm (utc) on Sep. 29, 2002]



 12:26 pm on Sep 29, 2002 (gmt 0)

Welcome to WebmasterWorld Monkscuba.
If you're looking for robots.txt info for beginners I'd suggest taking a look at Brett's Robots.txt Tutorial [searchengineworld.com] and then following the links at the bottom of that page to find some good related information.


 12:46 pm on Sep 29, 2002 (gmt 0)

Hello Monkscuba welcome to the board!

Robots Text is a seperate file on your server, which you will have to create. Everything you need to know gets explained if you follow the link Bill suggests.

>any comments on our site?

Sorry, Monkscuba, but that's something we generally do not encourage. Reviewing sites, pointing out sites, is for a variety of reasons something we do not do, wether per site in profile or per url drop.


 1:19 pm on Sep 29, 2002 (gmt 0)


Welcome to WebmasterWorld!

> Is it worth disallowing robots?

Why would you want to disallow them? It is Googlebot, after all, who gets you into the Google index!

Deciding which robots to allow and which to disallow is a complex decision, I suspect that most beginners (myself included!) should allow them all. Since that is the default behavior in the absence of a robots.txt file you do not need to do anything.


 1:24 pm on Sep 29, 2002 (gmt 0)

"Since that is the default behavior in the absence of a robots.txt file you do not need to do anything."

Thanks Mohamed_E - that's what I wanted to hear. I don't need to do anything. Good, because the tutorial talks about a lot of stuff I don't understand.

My mind is at rest and I can now carry on doing things I do understand like trying to get links and writing new pages.


 1:34 pm on Sep 29, 2002 (gmt 0)

>mind at rest
That's cool. Anyhow, when you have some traffic to your site it's really not such a bad idea to start controlling who can access which parts of your site.

Besides the fact that there are malvolent bots on the web, who can cause a lot of bandwidth, harvest all email addresses, render your logfiles to uselessness etc, you might want to restrict spiders to index parts of your site.
Do you have test pages up? do you have old files, which may cause duplication issues? what about your cgi folder? stats folders? You really want all bots to be able to index that?

If you should have any questions about the value of having robots text, as well as how to do it - feel free to ask!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved