Rapidly Changing, Dynamic Website

[vicyankees]

I have a site where we import information on a daily basis. Every night we add anywhere from 1-2k items while 1-2k items also expire and we have to take their page off the site. Any recommendations for the best way to handle with Google and the other search engines. I can't imagine they like seeing 1k pages come and go every day.

<Sorry, no specifics.
See Forum Charter [webmasterworld.com]>

[edited by: tedster at 6:48 am (utc) on Feb. 14, 2007]

[jonrichd]

You don't mention how long these items last on your site. If it's just a short time (< 2 weeks), you might consider putting them in a directory that's blocked from the spiders via robots.txt. Then, create pages organized by topic that are links to the specific items.

The thinking here is that if one of your major topic areas is blue widgets, that your list of blue widget items might get some search engine love, even if the items themselves aren't indexed. The same thing for your pages on red widgets and green widgets.

If the theme of your items stays consistent, but the content changes regularly due to changing conditions (something like a weather forecast for a specific city), then consider leaving the filename the same, and just update the content. You might also want to leave the title and headings the same to show the SEs that the page's theme hasn't changed.

If the items are around longer where you might get them indexed, and get referrals, I would make sure your 404 processing is working correctly, and that you have a custom 404 page so visitors can find similar related material should they click on an expired link.

[new_shoes]

"while 1-2k items also expire and we have to take their page off the site"

No chance of leaving them there, but listed as "expired"? Leave them greyed out, remove unnecessary details and add a big link to "active listings".

This would allow you to get dead content listed - arriving visitors would know immediately where to click to get fresh content. And, providing its original content, google will love it.

[trinorthlighting]

Yep,

That is what we do, list the items as sold out and put them in a discontinued item category. That way when people hit the items the click on to the fresh items.

[vicyankees]

I have custom 500s in place that display other relevant information to the users. The reason being 500s is because the database records no longer exist. Would I be better off creating a generic "noindex" page that gets served up instead of generating a 500?

I want to make sure Google is not going to penalize me for this large fluctuation. The "widgets" stay on the site for at least 30 days and usually about 90 days. I prefer to have them indexed by Google and the other Search Engines because we get some good long tail organic traffic.

[trinorthlighting]

Why not reuse the urls? Ever think of that?

[nippi]

I'd expect the url's can;t be reused, as they contain product name, if a rewrite rule is being used.

[vicyankees]

nippi is correct - i am using rewriting throughout and the id number of the widget is in the url.

[trinorthlighting]

Thats a big reason why our ecommerce sites do not use name, we use product numbers instead.

[vicyankees]

Our industry doesn't really allow for that - we customize the url based on the id of the property because the public can identify with this id and we use a primary characteristic of the item. Both of these being unique and never duplicative.

[grandpa]

the database records no longer exist

Could you create a supplemental database before you delete the records?

[vicyankees]

That's one theory I was exploring. I have a supplemental db and I had planned to do a check on the first page load - if the record doesn't exist, rather than load a 500 server error, take another pass against the supplemental db and if the record is valid there, pull a result of records from the first that are similar to the one that no longer exists with a message that this item is no longer available. But this is all from a users perspective. Should I throw up a 500 error or just the regular page with a noindex so google doesn't think I have server issues and just remove the page from its index.