Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: phranque
These pages would be generated dynamically with the part number in the title, description and keywords tag, along with part information.
The pages can be generated fairly easily, according the the developer, but getting them crawled and indexed is concerning me.
I've seen large sites with 700,000+ pages use a series of site maps to get it's pages indexed. But what do we do with a database of 60 million parts that my client wants indexed?
Any ideas would be appreciated. I hope this is the right catagory to post this question.
If these are genuine products the client sells you may also submit the whole thing to Froogle as a .csv dump.
What do you think? It's the number that I'm worried about. Crawling that many pages would take months, at an average 2 pg per second crawl rate.