XML Sitemap Generator for 1 million page site

looking for a solution

[RichardV]

I have a local search site for auto repair & services, there are 20 categories and 30k cities in the US and aprox 500k business profiles in our database. So about 1.1 million possible pages. The people at XML-Sitemaps.com told me that there is not an off the shelf software solution to generate a million page xml sitemap.

I know we can generate the site map and upload it 50k urls at a time but we're looking for a script that will maintain it for us.

Is anyone familiar with google's beta tool? [code.google.com...]

I'd appreciate any and all advice you guys have.
thanks in advance.

[lucy24]

Before you start tearing out your hair, stop and ask why you need a sitemap. You said "1.1 million possible pages". That word "possible" is a red flag. Or at least a yellow flag. I smell search results: exactly the kind of thing g keeps saying they don't want to index. (Whether this is in fact true is a whole nother question.)

Are all those 1.1 million pages genuinely different from each other? And, equally important, will the content they offer at the moment of crawling be the same as the content a human user sees, days or weeks later?

[RichardV]

There are 20 categories x 30k cites = 600k search pages. ie: "auto repair san jose" , "window tinting dallas" .
There are 500k unique business listings all with their own profile pages.

[edited by: phranque at 12:15 pm (utc) on Apr 28, 2013]
[edit reason] no personal urls please [/edit]

[TheOptimizationIdiot]

I would think as far as an XML sitemap goes you're in "custom" territory.

As far as uniqueness like Lucy24 was talking about goes, I understand your site will have unique pages on it, but the question I would ask is how are those going to be unique (besides the template) from what other people are already doing?

[RichardV]

Correct. The 500k business listing pages won't be worth anything to google because they are just name, address, phone, the same information all the local directory sites have on the businesses. I'm thinking maybe to only have my search pages indexed and businesses that fill out a unique profile (paid advertiser) with us. Thanks for your responses.