I have a bout 50,000 products in my database and I use asp/mysql for generating dynamic pages, all unique descriptions and keywords. The only problem is...it takes about 4-5 seconds for the pages to load, not because the page's filesize is too big but because of the server speed (I'm guessing) and the large database (50,000 records). Could that be the two factors that slow down google spidering those pages? I've had those pages up for about two weeks now and only about 800 got spidered so far. I see some of my competitors doing the same exact thing, but they have around 500,000 pages spidered by google and they started doing it at the same time I did. (they also have exact linking structure, backlinks to the site, age of the site, google pr and all the other little factors are the same). Their pages load instantly though, so should I get a faster server and reduce the database size to speed up google spidering the pages? Any ideas would be greatly appreciated.
In my experience, Google really does care about extremely slow server response, especially when it happens regularly and is not just once in a while. They don't want their users to get a poor experince from a results page they serve up.
Yes, I'd say do whatever you need to in order to tune the database functioning. But also, realize that with a relitively nerw set of pages, the new Google crawling pattern may take a while to get deep into the site.