I don't want to be listed yet

Is disabling anonomas access enough?

[currington]

I have a copy of a web site running on my testing server but I don't want it indexed as I don't want to pick up a bann for duplicate listings. I have disabled anonymous access for the server but is this enough? do I need to change my robots.txt file and my meta tags aswell?

Sorry if this is not in the correct forum.

[gilli]

Not sure about annon access. Do you mean a login is required? If so then I guess you are safe. Otherwise do the robots OR meta tags, I don't *think* you need both (I only use the meta tags), but I'm not sure what would happen if you have meta saying "index me" and robots.txt saying "stay out".

Check out [google.com ] and the whole webmaster section in general. You may find the answers there.

Another thing to note is that if you don't have any links pointing at your dev site GoogleBot won't find it.

Keep an eye on your logs to make sure everything is happy.

hth

[currington]

googlebot never apears in my logs, my logs are asp/db based but I don't think that is why.

I think I will change my robots.txt and the meta tags on the home page just incase.

I have no links to the testing server.

"HTH" <--- yes it did thank you.

[currington]

Ok,

I have read the link you gave me, thank you.

I have changed my meta tags on the home page to
<META NAME="ROBOTS" CONTENT="noINDEX,noFOLLOW">

And altered my robots.txt file to

User-agent: *
Disallow: /

But will this mean that I will not be able to list this site on my server in the future? or is it just a case of linking it to one of my other sites (Say my web dev site that is listed on google and such) then correcting the meta tags and robots.txt file?

[heini]

Currington, that robots.txt will prevent any bot accessing the site.
However, you say you disabled anonymous access - does that mean it's password protected? In that case bots obviously are blocked anyway.

[yetanotheruser]

currington,

In my experience, disallowing Googlebot in robots.txt doesn't upset him at all and he'll check it quite regularly.. Whenever we've put sites live and removed the robots.txt Googlebot seems to spider them within a week or two.

In short, robots.txt should IMHO cause you no problems when it comes to going live, and is probably your best option.

HTH :)

(arg.. thread moved! ;) )

[gilli]

googlebot never apears in my logs, my logs are asp/db based but I don't think that is why.

Do other user agents show up in your logs? It should just bee a matter of reading the request.useragent ([google.com ]).

As for the rest of the stuff, yetanotheruser is right as far as I know.

[currington]

---------------------
you say you disabled anonymous access - does that mean it's password protected?
--------------------
Yes it means that you have to have a user account on the web server to access it.

--------------------
In that case bots obviously are blocked anyway
--------------------
With all due respect that was not obvious else I wouldn't have asked.

When it comes to something as serious as the possibilty of picking up a bann or poor listing for the live site I can not take any chances, this is an ecommerce site that is generating a substancial turn over and I don't want to compromise that in any way.

---------------------
6: yetanotheruser 9:43 am on Mar. 5, 2003 (utc 0)
currington,
In my experience, disallowing Googlebot in robots.txt doesn't upset him at all and he'll check it quite regularly.. Whenever we've put sites live and removed the robots.txt Googlebot seems to spider them within a week or two.

In short, robots.txt should IMHO cause you no problems when it comes to going live, and is probably your best option.

HTH :)

---------------------

Thank you I have now done that and I can rest happy in the knowelege that my testing server will not screw up the live site.

Cheers for the help guys. :p