homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

G crawling and indexing a large site
My client has 60 million pages to get indexed in search engines

 3:56 am on Feb 15, 2005 (gmt 0)

A client of mine is a parts supplier with 60,000,000 parts they'd like to be indexed in search engines, particularly G.

These pages would be generated dynamically with the part number in the title, description and keywords tag, along with part information.

The pages can be generated fairly easily, according the the developer, but getting them crawled and indexed is concerning me.

I've seen large sites with 700,000+ pages use a series of site maps to get it's pages indexed. But what do we do with a database of 60 million parts that my client wants indexed?

Any ideas would be appreciated. I hope this is the right catagory to post this question.




 3:07 am on Feb 16, 2005 (gmt 0)

Google crawls links, just submit the main page, and all the links will ba added. I think:), That may have changed:)


 3:17 am on Feb 16, 2005 (gmt 0)

Presumably, there is some sort of category structure behind that many products? Just make sure that the category structure is crawlable and does not use drop-downs/javascript to advance to next level (or if it does that there is an alternative route).

If these are genuine products the client sells you may also submit the whole thing to Froogle as a .csv dump.


 3:25 am on Feb 16, 2005 (gmt 0)

Btw, don't expect to get all products into index if your home page PR is below 8 for that many products. You might want to group and expose only logical groups instead of individual products -- I bet they are all very similar and easily groupable with that total number.


 3:27 am on Feb 16, 2005 (gmt 0)

Froogle seems like a good idea, but what about the 60 Million items part of it? Would Froogle want a feed with 60 million part numbers?

What do you think? It's the number that I'm worried about. Crawling that many pages would take months, at an average 2 pg per second crawl rate.


 3:34 am on Feb 16, 2005 (gmt 0)

Ask froogle, I'm not qualified. And I doubt anybody here is. And think about grouping -- I doubt very much that each individual product really is that unique.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved