Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

G crawling and indexing a large site

My client has 60 million pages to get indexed in search engines



3:56 am on Feb 15, 2005 (gmt 0)

10+ Year Member

A client of mine is a parts supplier with 60,000,000 parts they'd like to be indexed in search engines, particularly G.

These pages would be generated dynamically with the part number in the title, description and keywords tag, along with part information.

The pages can be generated fairly easily, according the the developer, but getting them crawled and indexed is concerning me.

I've seen large sites with 700,000+ pages use a series of site maps to get it's pages indexed. But what do we do with a database of 60 million parts that my client wants indexed?

Any ideas would be appreciated. I hope this is the right catagory to post this question.



3:07 am on Feb 16, 2005 (gmt 0)

10+ Year Member

Google crawls links, just submit the main page, and all the links will ba added. I think:), That may have changed:)


3:17 am on Feb 16, 2005 (gmt 0)

10+ Year Member

Presumably, there is some sort of category structure behind that many products? Just make sure that the category structure is crawlable and does not use drop-downs/javascript to advance to next level (or if it does that there is an alternative route).

If these are genuine products the client sells you may also submit the whole thing to Froogle as a .csv dump.


3:25 am on Feb 16, 2005 (gmt 0)

10+ Year Member

Btw, don't expect to get all products into index if your home page PR is below 8 for that many products. You might want to group and expose only logical groups instead of individual products -- I bet they are all very similar and easily groupable with that total number.


3:27 am on Feb 16, 2005 (gmt 0)

10+ Year Member

Froogle seems like a good idea, but what about the 60 Million items part of it? Would Froogle want a feed with 60 million part numbers?

What do you think? It's the number that I'm worried about. Crawling that many pages would take months, at an average 2 pg per second crawl rate.


3:34 am on Feb 16, 2005 (gmt 0)

10+ Year Member

Ask froogle, I'm not qualified. And I doubt anybody here is. And think about grouping -- I doubt very much that each individual product really is that unique.

Featured Threads

Hot Threads This Week

Hot Threads This Month