homepage Welcome to WebmasterWorld Guest from 54.197.19.35
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

    
In the middle of creating a brand new site
How can I get google not to spider it?
johnnydequino




msg:51201
 8:00 pm on Jul 28, 2003 (gmt 0)

I am in the midst of creating about 60-70 new pages of content for a new domain name.

Once a page is finished, I would like to live it 'live', but I don't want google to spider it until the site is complete.

How do I make sure that google won't spider any pages until I am ready? Is it possible?

jd

 

dmorison




msg:51202
 8:56 am on Jul 29, 2003 (gmt 0)

Be careful.

If you use any of the standard exclusion techniques (robots.txt, meta tags etc.), then it is likely to be a very long time before robots come back to index your pages once you actually want them to.

Having said that, I think Googlebot checks robots.txt on a fairly regular basis, but either way I wouldn't rely on it.

I would either;

a) Not put them anywhere on the public Internet until you actually want them spidered

or

b) Use a completely different URL for your test version, and then change the URL to something that can be spidered once you want to have the content indexed.

dillonstars




msg:51203
 9:53 am on Jul 29, 2003 (gmt 0)

Having said that, I think Googlebot checks robots.txt on a fairly regular basis, but either way I wouldn't rely on it.

I recently stopped google visiting a site until it was finished by using the robots.txt file, and when i removed the block from the file it was only a matter of days before google started spidering. I did make sure that i had quite a lot of links to the site before I started allowing spiders though. This seemed to work, and I would make sure that you do start getting links while you are developing the site (if you can).

smokin




msg:51204
 10:03 am on Jul 29, 2003 (gmt 0)

Just put you test site 5 or 6 levels deep from the root and you should be ok.

tigger




msg:51205
 10:20 am on Jul 29, 2003 (gmt 0)

Yes I agree if I'm working on a large site thats what I do it works a treat

Dolemite




msg:51206
 10:56 am on Jul 29, 2003 (gmt 0)

How do I make sure that google won't spider any pages until I am ready?

Turn off the toolbar or use a different browser while you're in development.

Also, turn off indexes in .htaccess if you don't need them.

Add this line to .htaccess:

Options -Indexes

Or just add the -Indexes if you already have an Options directive. You can do this at root URL level if you want it throughout the site, or just to your development subdirectory by using the appropriate .htaccess file. This is more of a general tip for development...I don't think it will impede spidering necessarily, unless googlebot tries to crawl a directory. Anyway, it keeps people and bots/agents from seeing your files.

[edited by: Dolemite at 11:15 am (utc) on July 29, 2003]

GrinninGordon




msg:51207
 11:01 am on Jul 29, 2003 (gmt 0)

Run the site on internal not full URL path links (which will allow you to build the site based solely on using the server IP) and do not point the domain to the hoster's nameservers until ready.

johnnydequino




msg:51208
 2:47 pm on Jul 29, 2003 (gmt 0)

Thanks for all the tips.

I created the pages six pages deep, with nothing on the first level. Let's see how that works. If google picks up those pages, I would be surprised. =)

Johnny Dequino

coolasafanman




msg:51209
 5:39 pm on Jul 29, 2003 (gmt 0)

6 pages deep sounds like a lot of extra work. why not just put a password on the site?

jady




msg:51210
 5:45 pm on Jul 29, 2003 (gmt 0)

Never had problems with G indexing the site as long as its a new domain and no links are pointing to it. At most, have seen G only pick up Robots.txt and thrn run away.

Or just use a directory /new/index.html and it wont find it!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved