We currently have blocked googlebot access to our specific product/item level pages (aprox 13,000) to avoid any possible dup page penalty for those pages since many vary only slightly (less than 10%) and may be seen as duplicates. We have what we feel are legitimate independent product pages. e.g. a product line of essential oils say 30 items or so (we have aprox 15 essential oil lines.) The unique content within a product line is the flavor of the oil. We have vendor and product line content on each product page for efficient navigation. What varies is the ingredients, flavor and graphic (time stamp, meta tags etc. are all created to do the best we can to make them unique) but since in many cases the vendor and line information along with the navigation content is much more substantial then the unique content, we had decided to publish what we feel are the best content pages possible for our customers and not degrade them to satisfy spidering. However, with the advent of Froogle we feel that we must revisit this decision and are seeking guidance.
Our dilemma is that if we allow Google to spider our product pages we may get banned or penalized because of what we understand is Google's dup page criteria. According to Froogle feed instructions we must remove our blocking of Google to the product pages in the feed.
Is there any way to resolve this dilemma?
We would appreciate any thoughts, help etc. in resolving this.
Hi LaTimer. Your question is way over my head, but I do have one thought. I think Google doesn't penalize for dupes, it just chooses to list one of the dupe pages and ignores the other one. So this may not hurt your site too badly, since right now they're not getting spidered anyways. So maybe, if you remove the "no spider" tag, Google will list only 1 set of dupe pages, and you get better listings with Froogle.
If anyone can clarify this that'd be great - this is just how I understand this.