New store over 100k products - will Google index all urls?

[duffer77]

We are about to launch an online store with over 100k pieces of merchandise = over 100k url's.

My question is; Would it be better to populate let's say 3k urls a day or just populate all merchandise at one time.

I am wondering if google will index all url's if we do one time upload.

[trinorthlighting]

I would post them all at once, it will take time for Google to sort through them all anyways. The quicker you put them out there, the quicker they will index if they meet Google standards. Add to that, the sooner you get traffic, the sooner you might start generating sales and getting backlinks from shoppers.

[tedster]

I don't think that delaying the availability of all your URLs will speed indexing for you. You're right that all URLs are not likely to be included in the index, and especially not early on. But the most effective way to launch seems to be to publish it all and let Google's crawling and indexing logic prevail.

If you follow the pattern as it unfolds, you may well see that many URLs get crawled, but not all will be indexed - especially in the early days. As your site grows in popularity and attracts more backlinks and PageRank, then more pages get included.

[duffer77]

I see thank you guys. As of now we are going to launch the store under subdomain.domain.com/product_url/.
No id in the url hopefully will help to index a lot of urls.

Is there anything you were guys would suggest on doing after this is launched or before this launched?

[BradleyT]

Is there anything you were guys would suggest on doing after this is launched or before this launched?

Read the article linked in this post - [webmasterworld.com...]

Like tedster said - more links, more pagerank.

[duffer77]

Thanks for great article id did not have a chance on reading this yet.

[tedster]

All that - and be absolutely scrupulous about preventing canonical URL problems [webmasterworld.com] from day one.

[internetheaven]

We are about to launch an online store with over 100k pieces of merchandise = over 100k url's.

If this is a real store and the products are all yours with unique pages and descriptions then launching altogether should be fine as long as your internal linking is strong.

If this is an affiliate "store" where all the products and descriptions are the same as thousands of other affiliate stores then the number of URls are not your main concern in relation to getting indexed.

[Whitey]

There was some advice provided by Matt Cutts several years back on the release of URL's for large sites ... the exact link escapes me for the moment , but it was on a brief part of a MC YouTube clip.

Suffice to say the advice made no difference to indexing when i observed a large site index 100k's of pages going up. Do it all together or a step at a time .... no difference.

[pontifex]

it will index all pages, if you link them right - but the site: command will never return the real number!

you can see that for OLD big ecommerce sites. Amazon, ciao, ebay, etc. have millions of urls and show only a 10th or less with the site: command on google...

P!

[tedster]

some advice provided by Matt Cutts several years back on the release of URL's for large sites

One time was in 2006 (reference [mattcutts.com] when Microsoft had indexation trouble as tehy migrated pages on the Windows Live Writer blog from spaces.msn.com to spaces.live.com.

The url explosion triggered a flag and Google had to manually clear the flag. But that was a lot more than 100,000 urls - more like millions.

[duffer77]

It is an affiliate we ourselves do not sell merchandise. Unfortunately all data is provided by affiliate but i do not think they have a lot of solutions out there like we are coming up with.
We will do it at one time and see what happens. "Wish to know as much as Matt knows lol"

It is no longer important for us what site: brings "caffeine" update killed our search traffic we are looking to re-do the structure and goals on the site.

Store is an add-on will see what will happen.

Yap amazon or others at least do not have to worry about indexing or crawling.... and are sort of independent from google. Us small people we need google:(

[tedster]

Unfortunately all data is provided by affiliate...

Then you're up against being what Google calls a "thin affiliate". If there are other sites with the same exact information on their pages, you will probably have trouble getting your pages into the index.

i do not think they have a lot of solutions out there like we are coming up with.

Providing a unique and clear "added value" is exactly what it takes. Here's a good reference thread [webmasterworld.com].

[duffer77]

tedster,

This is really interesting. Hmmm sounds like i might have some extra work ahead of us. I am wondering how much the criteria have changed for so called "thin" sites and how to tackle this task the best.

Such as extra features those can be developed and added to bring extra value. I am not sure about 100k reviews such as description goes since that can become price'y and not worth of investing in it.

[tedster]

I am wondering how much the criteria have changed for so called "thin" sites

From everything I've heard lately, they are if anything more strict. Google feels that their users don't want to visit pages that are just like other pages. So an affiliate website needs to offer a strong unique value, over and above the affiliate purchases they offer.

Just a couple months ago, one of the Google reps said something like this - If your site is essentially an affiliate feed with just a few other things tacked on, then we don't want to offer it on our search results. But sites that do offer a unique value or service and also have affiliate links are not a problem.

That's just a paraphrase, but I think it's a fair representation.

[SEOPTI]

Unique content and Page Rank, Page Rank and again Page Rank, this is what you need. Not quantity of links, quality is the most important part mixed with some trust from older domains. Try to keep the quality PR links on topic.

[duffer77]

Looks like we are up to something challenging or just ......... lost the battle before we even started.

Sounds like google is really trying to promote their own products, slowly but surely moving and buying other companies and sending traffic to what they owe.

Imagine this; Optimized Search engine backed by their own companies that sell and make them profit. Maybe not just yet but who knows with their venture capital being invested in 100's of companies all over the world.

[tedster]

Sounds like google is really trying to promote their own products

I don't think that's what this particular guideline is all about - it really is about giving their end users a good experience. Search results with identical auto-generated information all the way down really were a bogus experience and they had to do something about it.

[duffer77]

I really hope this is not the way the G is heading...