How To Optimize For Large Retail Sites

How to optimise for 10,000+ products.

[foolsgold]

Having worked on many sites where the focus was very clear with a few services/products I am now looking to work with a site that has 10k plus products.

Are there some basic do's and don'ts I should be aware of?

[trinorthlighting]

We have a site that has a few thousand products. Make sure you have meta tags for each product line and the structure is good.

[tedster]

With a large number of products you can more easily end up with duplicate urls for the same exact content - especially if you have different ways of slicing and dicing the database - so be wary of that.

Also, consider ways to feature the most important products so that there is a shorter click path to those pages. Sometimes the drill-down path gets too long to keep googlebot (and PR circulation) happy.

Along those lines, give careful thought to products that are in more than one category or subcategory, again to avoid the duplicate url situation.

Beware of using a manufacturer-supplied description and nothing more on the page -- your page should not be identical to others on the web selling the same thing. The more you can customize and add value for your visitors, the better you can do.

[netmeg]

Make sure you have a way to give each product a unique title and meta description, otherwise they'll all be filtered.

[foolsgold]

Thanks all! Good call on the multiple entry points for porducts, now all I have to do is figure out a way to limit this to one, in search engine terms. - robots txt I guess.

Also, yes each product will have unique information - content and meta data.

What about links? We cannot get links for each speciific product so is it best to target best sellers or go for more generic terms and that the association (LSI?) will continue per product?

So, go for 'bathroom taps' instead of 'chrome plated bathroom taps'?

[digicam]

Can I ask what happens to items you delete from your site's database perhaps because they are no longer on sale.

Google would have indexed these items previously.

Do they just get dropped for example?

Does this create any problems for a ranking?

Are there any strategies to cope with this?

[trinorthlighting]

Are they 404 pages now?

[digicam]

It is just a general question about how other merchants handle the removal of products from their databases and how Google handles this - I expect they would be 404 I guess.

[trinorthlighting]

What we do, if the page is in the regular index, we typically change the page by adding a new product and taking down the old product. That way it seems to get indexed quicker.

If its in the supplemental index, we 404 the page and we do a 301 redirect to our home page when we take the product off. That way if it comes up in a query and the user clicks via google, the customer will be redirected to our home page instead of getting a 404 page.

[digicam]

So in general the removal of product page data and google is a serious issue that needs management - Is that correct?

I was just ignoring it and deleting a product as needed.

I cannot do your 1st suggestion ie product page reuse, does the second suggestion involve setting up a new redired manually for each deleted product - if so I cannot do that either.

any other suggestions?
cheers.

[trinorthlighting]

The reuse is nice because if it is in the regular index and has PR already, reusing it keeps the url PR in most cases.

If you can not, redirect the url to a similar product or to your homepage.

You can also use the google url removal tool as well.

[digicam]

Hi, I have a couple of largish sites and products go out and new ones come in daily so I am wondering about the wisdom of deleting the old ones, I cannot reuse the same page and I do not have time to create seperate 404s

All I can think of is to not delete old products but change to "not available" - which I hate doing.

any other thoughts?

[Simsi]

Following up on the database serving, a dupe content question: if you serve "category" pages and for example 5 products per page with short descriptions, if the "black products" page contains the same products (and descriptions) that also appear on the "tall products" page, even though META data is different, I assume that will be considered dupe content?

Assuming "yes", if you include a 100 word description of the category, and randomise the products a little, will this avoid that? If not, how does one get round this with a big database?

Cheers

Simsi

[tedster]

if you include a 100 word description of the category

Not a 100 word description (too long) but a 150 character description would do the trick. Think about the snippet that should appear in the search result if that url is returned. Google does have a disturbing habit of making a "quick decision" about duplicate filtering based only on the title and meta description elements, so this can be important.

[tedster]

I do not have time to create seperate 404s

If the url no longer resolves, then your server should automatically generate a default 404. That shouldn't take any time from you other than deleting the product. With a little more effort, you can create a "custom 404" page that gives visitors a few good alternatives.

[andye]

Interestingly, Amazon has started putting keywords into their URLs - presumably this is for search engine purposes.

E.g. some of their URLs now look like:

example.com
/
ProductType-Unusual-Words-From-Title-Eg-Widget-Green-Shiny
/
product SKU and other stuff... etc....

hth, a.

[mojomike]

I have a simple question in relevance to to large web sites with lot's of products.

what is the amount of content change needed to avoid a penalty of any sorts?

[Simsi]

Tedster:

Not a 100 word description (too long) but a 150 character description would do the trick. Think about the snippet that should appear in the search result if that url is returned. Google does have a disturbing habit of making a "quick decision" about duplicate filtering based only on the title and meta description elements, so this can be important.

Thanks tedster...so you don't think the same product descriptions appearing on both pages is an issue then if this snippet and the META tags are different?

[blend27]

When the product goes out of stock, or gets droped all together, we replace the BUY Button with the text link to the main category of the widget. The text link is usualy 4 words that used to be in the ALT Tag of the BUY Button.

The link to the product from main category page gets moved to SOLD OUT ITEMS PAGE, to the buttom of the display, then when widget page gets reindexed we add links from within the copy of the widget to newly added widgets on the subject.

this way new pages get indexed faster and have a backup links from already established pages.

if the product is back in stock everithing goes to the way it was: DOMINATING THE SERP

:)

[iblaine]

Ensure your site is crawlable. With 10k products you can make it difficult for SE's to find your products. Create a good sitemap & use tools like Google Sitemaps to submit your site.

[Lorel]

You also need to be very careful with similar products, for instance table chairs and lawn chairs. Even though they are totally different products with different titles and descriptions, etc Google tags one as supplemental on a site I manage.

[davidof]

> Interestingly, Amazon has started putting keywords into their URLs - presumably this is for search engine purposes.

Yes it is, they recently hired some external SEO consultants and it was one of a number of things that was suggested.

Also don't 301 out of stock items but keep the page and suggest alternatives in the same category.

[Robert Charlton]

With a large number of products you can more easily end up with duplicate urls for the same exact content - especially if you have different ways of slicing and dicing the database - so be wary of that.

Here are some thoughts on this I'd posted a few years ago... doesn't apply to all sites, but applicable to many...

How to make a form driven site searchable
[webmasterworld.com...]

[pageoneresults]

Taxonomy. Architecture. Foundation.

Be sure you have the foundation to support such a large undertaking. Lots of solid planning now will minimize the number of changes you have to make a year or two from now to accommodate growth.

Check, double check and triple check your work. Especially in a rewrite environment. Make sure the proper server headers are being returned at each step of the way. Hack your way through the URI string and just triple check all of that.

Create category index pages that act more like site maps. Keep those pages well linked to. If you were to draw your site out on a white board, what paths does the spider take to get to the content? Make sure you provide both horizontal, vertical and diagonal link paths. ;)

[spikey]

We don't remove products. We show that they're discontinued and suggest an alternative. No sense throwing out all of that unique content and page aging. Plus, you just might make a sale from a customer who would have otherwise gone elsewhere.

[avibodha]

And make sure you're not serving session id's in the URL to the search engines.

[tedster]

so you don't think the same product descriptions appearing on both pages is an issue then if this snippet and the META tags are different?

Not exactly. Without the unique titles and descriptions you often can't even get to square one - it's right to Supplemental with your urls -- if you're lucky enough to get indexed. But even with unique titles and descriptions, nearly duplicate body content can still trip you up. Yes, it takes extra work to ensure distinct content but it really pays off.

[andye]

Interestingly, Amazon has started putting keywords into their URLs - presumably this is for search engine purposes.

Yes it is, they recently hired some external SEO consultants and it was one of a number of things that was suggested.

davidof, that's very interesting! Any idea what other changes they made?

Best wishes, a.

[ukDevelopments]

Tedster - Beware of using a manufacturer-supplied description and nothing more on the page -- your page should not be identical to others on the web selling the same thing. The more you can customize and add value for your visitors, the better you can do.

Very good advice.

[elgumbo]

I never delete any products. If the page is in the index then IMO it's better to keep it there and either add a link to similar products or show a "we're sorry but this product is not available. It may however be available on one of these other [adsense] sites." ;)

Also, it's worth checking your stats to see how often an old stock item page is visited. If it's a lot, then maybe you should be getting some more of that item in stock quickly.