Welcome to WebmasterWorld Guest from 54.163.23.73

Message Too Old, No Replies

Will the first time google crawls my site matter more than other times?

     

Planet13

8:48 pm on Aug 29, 2010 (gmt 0)

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



Hi there, Everyone:

I have a new e commerce site in development that will EVENTUALLY have about 12 different categories and about 180 products.

Should I block the site with robots.txt until ALL the products and categories are up in place?

Or is it ok to start letting search engines index the site if I only have two categories and maybe 15 products up, and gradually add categories and products as they become available? (there won't be any broken links or pages with minimal text while doing this).

In short, does the first crawl / indexing of a new web site have any more significance than subsequent crawling and indexing?

limoshawn

3:15 am on Aug 30, 2010 (gmt 0)

5+ Year Member



I've done it both ways and have not really noticed any real difference as far as ranking. One thing is on the sites that I have blocked and then opened up with significant content is that it takes upwards of a month for google to get through the entire site. based on that it might be best if you are putting your best products in first to go ahead and let google at it so you can get those products indexed as soon as possible.

Planet13

3:27 am on Aug 30, 2010 (gmt 0)

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



One thing is on the sites that I have blocked and then opened up with significant content is that it takes upwards of a month for google to get through the entire site.


Yeah, now that you mention it, I remember that happening to another site I had done a while back...

lets_learn

6:12 am on Aug 30, 2010 (gmt 0)

5+ Year Member



I do not have any hard facts to back my opinion but it's unlikely that Google would like to "profile" a site and keep grudges based on its first crawl.