Google and large datafeed sites

[majestic12]

Hi there,


I have 2 basic datafeed questions I would be grateful for some answers on...



I have a custom datafeed with 30,000 products which will be replaced "weekly"....



1. Can I upload 30,000 pages in one go...


2. The end of the week comes, I need to replace the feed. Do I delete the previous 30,000 pages and upload the new or do I add the new 30,0000 and remove the old 30,000 a week later.?

Now when it comes to datafeeds I am a newbie, as for SEO etc I am not... Now my standard SEO knoweldge tells me uploading 30,000 pages is a "no no", the thing is though the big players surely have to upload their "new" products on a frequent basis..

How do we best handle large datafeeds that need to be updaated regularly, I do not want to leave 1000s of pages of outdated products on my site..

Would be grateful for some advice...

Many thanks...

[LifeinAsia]

Hmmm, not sure if this is actually a database question. Are you talking about uploading your products to Google Base? Or are you talking about product pages on your site and wanting to upload site maps to Google? Or...?

[majestic12]

Hi there,

I am talking about uploading a datafeed to a site, then building pages from the feed within site..so I suppose I am using the feed as a database..

My worry is will google be happy if I create 30000 pages from my feed, the big online retailers must update their products all the time. I just wasn't sure if google would not like this... I have only ever built 200 new pages per day, plus my feed will need to be replaced about once per week...

Regards.. David

[tedster]

I assume each new product also has its own unique URL. Is that correct? And how about the actual product description - will that be unique to your site or duplicated on other sites?

[majestic12]

Hi there,

Yes each url will be unique and the content will be 50% unique...

[tedster]

If you change the content of 30,000 URLs every week but keep the URL the same, then your search traffic will usually be poorly targeted and not buy - because what they searched for is no longer there.

If you make 30,000 URLs disappear every week but put up 30,000 new URLs with new content, you will have almost no time where those URLs are findable through search for they're content. Not only that, but you'll rarely develop any legitimate backlinks.

If you hope to use search traffic as a source of revenue, then I'd re-think the entire business model. Something needs to stay stable for a longer period of time.

[majestic12]

Yes I did wonder about that, I thought though I could do this...

Add 10000 pages week 1
add 10000 pages week 2
add 10000 pages week 3
add 10000 pages week 4 "and" remove week 1's 10000 pages
add 10000 pages week 5 and remove weeks 2's etc..

This way I would always have 30000 live and be filtering out the old deals...

Yes maybe also a "large" stable section of pages may be good as an anchor,,

Not sure how google would react though, the thing is surely the large online stores need to be constantly adding and removing in some fashion..

btw thanks for all yout input...

Regards David

[futureX]

I think something needs to be clarified really, it sounds to me like you're uploading 30,000 pages about products then removing them a week later to upload a different 30,000?

So any products on your site are never there for more than a week? Even in the scenario you sated above there would still be barely enough time to build any link popularity or good rankings within the 3 weeks the information is up... It seems like a lot of temporary data to be cycling through, I wonder what value it is to the customer if this many products were only up for a week or two...

Not only that, but unless you've got some pretty hefty PR/Authority already google will most likely not index 10,000 pages a week.

One of the other problems google wise is how you will handle requests to products that no longer exist, once you've taken it down you will either have to 404 it which will cause google to dislike your site (and rightly so) or you will have to redirect the pages causing it to show as duplicate content and google won't like that either.

If you're expecting to get a lot of traffic from google I wouldn't count on it to move the business forwards. I think the main problem though is folks here can't get their head around what business model revolves around having 30,000 new products a week...

[majestic12]

Hi future x

The products are actually mobile phone deals that are updated 1-2 weekly by the merchant. Each handset has about 50 variations on the deal that is being offered, these are different monthly minutes, variations in message packages and also internet coverage.. As for removal I would do this, upload each batch of 10,000 products under a set category [ I am using wordpress], then when I am finished with them remove that particualr directory via webmaster tools...

If I were trying to sell computers etc each product would probabaly have a longer life span, the difficult thing is mobile deals change very frequently..

I quite often come across affiliate pages that are months out of date, maybe I am striving too much for perfection?

Maybe on reflection I carry out some sort of monthly staggered rotations..

Many thanks for your reply..

David

[James2]

How's it possible to upload them all so quickly, especially if your altering the content on the items.

[HuskyPup]

I was about to ask what widget could generate 30,000 new products a week...now I know!

maybe I am striving too much for perfection?

I feel so, is it any wonder most people cannot comprehend the deals if they are changing 1,560,000 times in a year?

I'm assuming that your intention is/was to promote these deals through the SERPs as opposed to an AdWords style of advertising? Were you considering keeping your index page relatively static and everything changing in the background or try and promote through a continually changing index page?

You'll be up against some very well-established and tough competitors who are already well-ensconced in the SERPs, are you sure about this?

[indyank]

majestic12 - See whether this will work out for you.

Have one unique url for each product (mobile phone deal) and keep updating it with the current deal while archiving the older ones. For example, if there is a "<brandname> mobile phone deal", I will have one unique url for it, say, with the slug "brandname-phone-deal". I will always updated this page with the current deal.On expiry, this will have a slug say "brandname-phone-deal-2009".

[majestic12]

Thanks for all the replies..

As for why so many products, the mobile companies in the UK offer hundreds of variations for each deal/handset.. one phone could be sold in 100 different packages.... it may be my hopes are totally unrealistic... the pages will be datafeed driven,,, one page for each product.. maybe I will just have to build smaller sites and slowy remove old product pages..

Again thanks for all the input..

[futureX]

I see, it makes more sense once it's put in some context. I can see the possibility to have many many pages from a few "deals". As I know how mobile companies can be with their deals and stock I won't weigh in on how better to present that data.

There's probably a lot of planning and frustration ahead, good luck :)

[majestic12]

Thanks FX..

Regards David