homepage Welcome to WebmasterWorld Guest from 54.211.70.79
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Local Search
Forum Library, Charter, Moderators: anallawalla & bakedjake & inbound

Local Search Forum

    
Starting a site with millions of pages (built on APIs)
dirkules




msg:4415114
 4:46 pm on Feb 7, 2012 (gmt 0)

Hey guys,

I'm in the process of developing a site that will probably have millions of pages, thanks to APIs with millions of entities. Most definitely about 99% of the content (companies/services/products/places) will already be published somewhere on the web.
Will I run into massive duplicate content issues? Even if I combine the content from different sources so it won't look like an exact 1:1 copy?

Now just concerning the business listings and from a search engine view, the site drills down like:
    homepage->state->city->category->business
    homepage->category->business (tens or hundreds of thousands businesses with huge pagination in this case)

and of course there's a search form, too.

I assume I should use noindex,follow for the second one?
Should I list all its categories on the business listings page? If so, should I also link them back to the categories? Not sure about the link juice here.

[edited by: tedster at 6:06 pm (utc) on Feb 7, 2012]
[edit reason] moved from another location [/edit]

 

anallawalla




msg:4415154
 7:07 pm on Feb 7, 2012 (gmt 0)

I have worked with some very large national directories and can offer this - Google will take some time to index it all (depends on how many millions), so a good sitemap.xml index method will help to get the more important pages indexed first.

Search engine visitors will come via category based phrases, so you don't want to block that path. I don't see a need for two hierarchies. The eventual business profile page should have only one instance.

You may have omitted one step - "businesses", which lead to individual "business" listings.

Directory users (direct visits) will use the internal search primarily and might browse geographically. You may want to consider how and where to use noindex,follow.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Local Search
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved