Welcome to WebmasterWorld Guest from 54.221.87.97

Forum Moderators: open

Message Too Old, No Replies

In the middle of creating a brand new site

How can I get google not to spider it?

     

johnnydequino

8:00 pm on Jul 28, 2003 (gmt 0)

10+ Year Member



I am in the midst of creating about 60-70 new pages of content for a new domain name.

Once a page is finished, I would like to live it 'live', but I don't want google to spider it until the site is complete.

How do I make sure that google won't spider any pages until I am ready? Is it possible?

jd

dmorison

8:56 am on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Be careful.

If you use any of the standard exclusion techniques (robots.txt, meta tags etc.), then it is likely to be a very long time before robots come back to index your pages once you actually want them to.

Having said that, I think Googlebot checks robots.txt on a fairly regular basis, but either way I wouldn't rely on it.

I would either;

a) Not put them anywhere on the public Internet until you actually want them spidered

or

b) Use a completely different URL for your test version, and then change the URL to something that can be spidered once you want to have the content indexed.

dillonstars

9:53 am on Jul 29, 2003 (gmt 0)

10+ Year Member



Having said that, I think Googlebot checks robots.txt on a fairly regular basis, but either way I wouldn't rely on it.

I recently stopped google visiting a site until it was finished by using the robots.txt file, and when i removed the block from the file it was only a matter of days before google started spidering. I did make sure that i had quite a lot of links to the site before I started allowing spiders though. This seemed to work, and I would make sure that you do start getting links while you are developing the site (if you can).

smokin

10:03 am on Jul 29, 2003 (gmt 0)

10+ Year Member



Just put you test site 5 or 6 levels deep from the root and you should be ok.

tigger

10:20 am on Jul 29, 2003 (gmt 0)

WebmasterWorld Senior Member tigger is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Yes I agree if I'm working on a large site thats what I do it works a treat

Dolemite

10:56 am on Jul 29, 2003 (gmt 0)

10+ Year Member



How do I make sure that google won't spider any pages until I am ready?

Turn off the toolbar or use a different browser while you're in development.

Also, turn off indexes in .htaccess if you don't need them.

Add this line to .htaccess:

Options -Indexes

Or just add the

-Indexes
if you already have an
Options
directive. You can do this at root URL level if you want it throughout the site, or just to your development subdirectory by using the appropriate .htaccess file. This is more of a general tip for development...I don't think it will impede spidering necessarily, unless googlebot tries to crawl a directory. Anyway, it keeps people and bots/agents from seeing your files.

[edited by: Dolemite at 11:15 am (utc) on July 29, 2003]

GrinninGordon

11:01 am on Jul 29, 2003 (gmt 0)



Run the site on internal not full URL path links (which will allow you to build the site based solely on using the server IP) and do not point the domain to the hoster's nameservers until ready.

johnnydequino

2:47 pm on Jul 29, 2003 (gmt 0)

10+ Year Member



Thanks for all the tips.

I created the pages six pages deep, with nothing on the first level. Let's see how that works. If google picks up those pages, I would be surprised. =)

Johnny Dequino

coolasafanman

5:39 pm on Jul 29, 2003 (gmt 0)

10+ Year Member



6 pages deep sounds like a lot of extra work. why not just put a password on the site?

jady

5:45 pm on Jul 29, 2003 (gmt 0)

10+ Year Member



Never had problems with G indexing the site as long as its a new domain and no links are pointing to it. At most, have seen G only pick up Robots.txt and thrn run away.

Or just use a directory /new/index.html and it wont find it!