Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
As a complete amateur, our web site may look a bit shabby, but right now we can't afford fancy web design. However, largely thanks to this Wemaster site, we get high rankings for our keywords, and I am still trying to optimise. The site has only been online since June, and I am gathering links and hopefully customers!
Anyway, I am very confused about all this robots.txt talk, and want to know if we have one or not! NO and we don't need one!I think it's the bit which says :
<meta name="robots" content="index,follow">
Yes? NO! That's the robots meta tag. Wow, you learn more every day!
If I want to disallow robots, where does this code go? Under the previous line? Is it worth disallowing robots?
Any pros got any comments on our site? Am I doing anything really wrong? Our market is very competitive on the ground, but most of the competing websites are not so great, and a PR of 5 is about as high as you'll find in our field (we are currently PR 4).
Sorry to bother y'all like this.
Have a nice day!
Hey! I can't see my homepage on my profile! How can anyone help me!
[edited by: Monkscuba at 1:26 pm (utc) on Sep. 29, 2002]
Robots Text is a seperate file on your server, which you will have to create. Everything you need to know gets explained if you follow the link Bill suggests.
>any comments on our site?
Sorry, Monkscuba, but that's something we generally do not encourage. Reviewing sites, pointing out sites, is for a variety of reasons something we do not do, wether per site in profile or per url drop.
Welcome to WebmasterWorld!
> Is it worth disallowing robots?
Why would you want to disallow them? It is Googlebot, after all, who gets you into the Google index!
Deciding which robots to allow and which to disallow is a complex decision, I suspect that most beginners (myself included!) should allow them all. Since that is the default behavior in the absence of a robots.txt file you do not need to do anything.
My mind is at rest and I can now carry on doing things I do understand like trying to get links and writing new pages.
Besides the fact that there are malvolent bots on the web, who can cause a lot of bandwidth, harvest all email addresses, render your logfiles to uselessness etc, you might want to restrict spiders to index parts of your site.
Do you have test pages up? do you have old files, which may cause duplication issues? what about your cgi folder? stats folders? You really want all bots to be able to index that?
If you should have any questions about the value of having robots text, as well as how to do it - feel free to ask!