homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

XML Sitemap Generator for 1 million page site
looking for a solution

Msg#: 4568533 posted 4:14 pm on Apr 27, 2013 (gmt 0)

I have a local search site for auto repair & services, there are 20 categories and 30k cities in the US and aprox 500k business profiles in our database. So about 1.1 million possible pages. The people at XML-Sitemaps.com told me that there is not an off the shelf software solution to generate a million page xml sitemap.

I know we can generate the site map and upload it 50k urls at a time but we're looking for a script that will maintain it for us.

Is anyone familiar with google's beta tool? https://code.google.com/p/googlesitemapgenerator/

I'd appreciate any and all advice you guys have.
thanks in advance.



WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

Msg#: 4568533 posted 9:40 pm on Apr 27, 2013 (gmt 0)

Before you start tearing out your hair, stop and ask why you need a sitemap. You said "1.1 million possible pages". That word "possible" is a red flag. Or at least a yellow flag. I smell search results: exactly the kind of thing g keeps saying they don't want to index. (Whether this is in fact true is a whole nother question.)

Are all those 1.1 million pages genuinely different from each other? And, equally important, will the content they offer at the moment of crawling be the same as the content a human user sees, days or weeks later?


Msg#: 4568533 posted 9:17 am on Apr 28, 2013 (gmt 0)

There are 20 categories x 30k cites = 600k search pages. ie: "auto repair san jose" , "window tinting dallas" .
There are 500k unique business listings all with their own profile pages.

[edited by: phranque at 12:15 pm (utc) on Apr 28, 2013]
[edit reason] no personal urls please [/edit]


Msg#: 4568533 posted 4:47 pm on Apr 28, 2013 (gmt 0)

I would think as far as an XML sitemap goes you're in "custom" territory.

As far as uniqueness like Lucy24 was talking about goes, I understand your site will have unique pages on it, but the question I would ask is how are those going to be unique (besides the template) from what other people are already doing?


Msg#: 4568533 posted 4:59 pm on Apr 28, 2013 (gmt 0)

Correct. The 500k business listing pages won't be worth anything to google because they are just name, address, phone, the same information all the local directory sites have on the businesses. I'm thinking maybe to only have my search pages indexed and businesses that fill out a unique profile (paid advertiser) with us. Thanks for your responses.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved