Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderators: goodroi

Beginners guide to robots

Help needed!

   
10:43 am on Sep 29, 2002 (gmt 0)

10+ Year Member



Dear webmasters,

As a complete amateur, our web site may look a bit shabby, but right now we can't afford fancy web design. However, largely thanks to this Wemaster site, we get high rankings for our keywords, and I am still trying to optimise. The site has only been online since June, and I am gathering links and hopefully customers!

Anyway, I am very confused about all this robots.txt talk, and want to know if we have one or not! NO and we don't need one!I think it's the bit which says :

<meta name="robots" content="index,follow">

Yes? NO! That's the robots meta tag. Wow, you learn more every day!

If I want to disallow robots, where does this code go? Under the previous line? Is it worth disallowing robots?

Any pros got any comments on our site? Am I doing anything really wrong? Our market is very competitive on the ground, but most of the competing websites are not so great, and a PR of 5 is about as high as you'll find in our field (we are currently PR 4).

Sorry to bother y'all like this.

Have a nice day!

Hey! I can't see my homepage on my profile! How can anyone help me!

[edited by: Monkscuba at 1:26 pm (utc) on Sep. 29, 2002]

12:26 pm on Sep 29, 2002 (gmt 0)

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Welcome to WebmasterWorld Monkscuba.
If you're looking for robots.txt info for beginners I'd suggest taking a look at Brett's Robots.txt Tutorial [searchengineworld.com] and then following the links at the bottom of that page to find some good related information.
12:46 pm on Sep 29, 2002 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hello Monkscuba welcome to the board!

Robots Text is a seperate file on your server, which you will have to create. Everything you need to know gets explained if you follow the link Bill suggests.

>any comments on our site?

Sorry, Monkscuba, but that's something we generally do not encourage. Reviewing sites, pointing out sites, is for a variety of reasons something we do not do, wether per site in profile or per url drop.

1:19 pm on Sep 29, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Monkscuba,

Welcome to WebmasterWorld!

> Is it worth disallowing robots?

Why would you want to disallow them? It is Googlebot, after all, who gets you into the Google index!

Deciding which robots to allow and which to disallow is a complex decision, I suspect that most beginners (myself included!) should allow them all. Since that is the default behavior in the absence of a robots.txt file you do not need to do anything.

1:24 pm on Sep 29, 2002 (gmt 0)

10+ Year Member



"Since that is the default behavior in the absence of a robots.txt file you do not need to do anything."

Thanks Mohamed_E - that's what I wanted to hear. I don't need to do anything. Good, because the tutorial talks about a lot of stuff I don't understand.

My mind is at rest and I can now carry on doing things I do understand like trying to get links and writing new pages.

1:34 pm on Sep 29, 2002 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member



>mind at rest
That's cool. Anyhow, when you have some traffic to your site it's really not such a bad idea to start controlling who can access which parts of your site.

Besides the fact that there are malvolent bots on the web, who can cause a lot of bandwidth, harvest all email addresses, render your logfiles to uselessness etc, you might want to restrict spiders to index parts of your site.
Do you have test pages up? do you have old files, which may cause duplication issues? what about your cgi folder? stats folders? You really want all bots to be able to index that?

If you should have any questions about the value of having robots text, as well as how to do it - feel free to ask!

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month