homepage Welcome to WebmasterWorld Guest from 54.242.18.190
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
I don't want to be listed yet
Is disabling anonomas access enough?
currington




msg:1525786
 8:48 am on Mar 5, 2003 (gmt 0)

I have a copy of a web site running on my testing server but I don't want it indexed as I don't want to pick up a bann for duplicate listings. I have disabled anonymous access for the server but is this enough? do I need to change my robots.txt file and my meta tags aswell?

Sorry if this is not in the correct forum.

 

gilli




msg:1525787
 9:19 am on Mar 5, 2003 (gmt 0)

Not sure about annon access. Do you mean a login is required? If so then I guess you are safe. Otherwise do the robots OR meta tags, I don't *think* you need both (I only use the meta tags), but I'm not sure what would happen if you have meta saying "index me" and robots.txt saying "stay out".

Check out [google.com ] and the whole webmaster section in general. You may find the answers there.

Another thing to note is that if you don't have any links pointing at your dev site GoogleBot won't find it.

Keep an eye on your logs to make sure everything is happy.

hth

currington




msg:1525788
 9:24 am on Mar 5, 2003 (gmt 0)

googlebot never apears in my logs, my logs are asp/db based but I don't think that is why.

I think I will change my robots.txt and the meta tags on the home page just incase.

I have no links to the testing server.

"HTH" <--- yes it did thank you.

currington




msg:1525789
 9:34 am on Mar 5, 2003 (gmt 0)

Ok,

I have read the link you gave me, thank you.

I have changed my meta tags on the home page to
<META NAME="ROBOTS" CONTENT="noINDEX,noFOLLOW">

And altered my robots.txt file to

User-agent: *
Disallow: /

But will this mean that I will not be able to list this site on my server in the future? or is it just a case of linking it to one of my other sites (Say my web dev site that is listed on google and such) then correcting the meta tags and robots.txt file?

heini




msg:1525790
 9:42 am on Mar 5, 2003 (gmt 0)

Currington, that robots.txt will prevent any bot accessing the site.
However, you say you disabled anonymous access - does that mean it's password protected? In that case bots obviously are blocked anyway.

yetanotheruser




msg:1525791
 9:43 am on Mar 5, 2003 (gmt 0)

currington,

In my experience, disallowing Googlebot in robots.txt doesn't upset him at all and he'll check it quite regularly.. Whenever we've put sites live and removed the robots.txt Googlebot seems to spider them within a week or two.

In short, robots.txt should IMHO cause you no problems when it comes to going live, and is probably your best option.

HTH :)

(arg.. thread moved! ;) )

gilli




msg:1525792
 9:49 am on Mar 5, 2003 (gmt 0)

googlebot never apears in my logs, my logs are asp/db based but I don't think that is why.

Do other user agents show up in your logs? It should just bee a matter of reading the request.useragent ([google.com ]).

As for the rest of the stuff, yetanotheruser is right as far as I know.

currington




msg:1525793
 9:53 am on Mar 5, 2003 (gmt 0)

---------------------
you say you disabled anonymous access - does that mean it's password protected?
--------------------
Yes it means that you have to have a user account on the web server to access it.

--------------------
In that case bots obviously are blocked anyway
--------------------
With all due respect that was not obvious else I wouldn't have asked.

When it comes to something as serious as the possibilty of picking up a bann or poor listing for the live site I can not take any chances, this is an ecommerce site that is generating a substancial turn over and I don't want to compromise that in any way.

---------------------
6: yetanotheruser 9:43 am on Mar. 5, 2003 (utc 0)
currington,
In my experience, disallowing Googlebot in robots.txt doesn't upset him at all and he'll check it quite regularly.. Whenever we've put sites live and removed the robots.txt Googlebot seems to spider them within a week or two.

In short, robots.txt should IMHO cause you no problems when it comes to going live, and is probably your best option.

HTH :)

---------------------

Thank you I have now done that and I can rest happy in the knowelege that my testing server will not screw up the live site.

Cheers for the help guys. :p

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved