How many seconds googlebot can wait for a page to load?

What about long SQL queries?

[tabish]

Hello All,

I wonder that how many seconds googlebot wait for a page to load? Suppose one of my dynamic page takes 20 seconds to execute a Mysql query, then will googlebot wait for that page?

Regards
Tabish

[rekitty]

Googlebot dynamically adjusts its crawl rate based on the response time of your site. In most cases it will never request the next page until the previous request is complete.

20 seconds? Dude, you need an index on that table!

[inbound]

We found out a while ago (due to an error which was quickly fixed) that Googlebot will wait more than 30 seconds for a page to load, and adjusts its crawl rate based on response times.

Similarly, if you have loads of pages, importance and serve pages in under a second Google will happily eat 80,000+ pages a day ;o)

[trinorthlighting]

I agree with inbound, we had an error as well that was taking about 30 second and googlebot still indexed it and had a 200 code to it. We have fixed it, but I can attest to 30 seconds as well.

[LifeinAsia]

Googlebot may wait 20 seconds for a page to load, but the more important question you should be asking is if your USERS will wait that long.

Secondly, you should be trying to figure out how to do things to get a much faster query. That is WAY too long for a public page.

[tabish]

My site is completely Database driven site and thousands of pages. Sometimes when i get much hits, my sql quaries takes much time to execute. I have optimized everything and made it As simple as possible but so many of my pages still takes 3 to 4 seconds to generate.

Shoudl i ask for my server upgrade, like more RAM or something like that?

[LifeinAsia]

Are you on shared hosting or a dedicated server? If shared, move to dedicated ASAP- it sounds like other users are slowing your site down.

BTW, 3-4 seconds is a lot different from the 20+ seconds mentioned in your original post; 3-4 seconds is not all that bad. Chances are you can always find more things to optimize. Are your tables indexed properly? Are all your tables fully normalized (if so, often you can optimize things by de-normalizing tables)? Does MYSQL have query caching?

[Frederic1]

Use a cache.

Check the following php functions :
ob_start()
ob_get_contents()
ob_end_flush()

[tabish]

I have a Dedicaed Server, yes, MySql Query cache is there.

I made a cache programme for my site, but then my server was storing too much cache files in the cache directories and file numbers crossed the limits and I ran out of Server Node.. then i deleted cache system.

Bytheway, my site has good hits.

Any other idea?

Regards

[LifeinAsia]

I don't know about MYSQL's query caching system, but you don't need (or want) to cache every single query- just the most commonly called ones.

Looking into upgrading your server's memory and/or processing speed may also work- you'll need to do some investigation to see if either (or both) is causing a bottleneck currently. Splitting the DB server to its own server may also be needed- it all depends on how your server is handling the current load.

But I also made some other suggestions- look into further optimization before throwing money at hardware.

[inbound]

Running well-optimised MySQL queries on a dedicated server should not cause pages to take 3 to 4 seconds. You need a load of optimised DB calls to be getting anywhere near that.

Are you sure that you have implimented indexes properly? It could also be that your joins are overly complicated. There are many reasons why MySQL is running slowly.

You may want to search for 'mysql performance blog' on Google, it'll give you a few pointers. It'll also give you inspiration to dig deeper into specific problems.

A lot can depend on how strictly you follow normalisation rules, sticking to them 100% is not always best. An example of this is a 170,000 page dynamic site that I've just completed DB optimisation on. I wrote a program to 'flatten' the data from a properly structured source DB every evening. The flattened DB is larger than the normalised DB but, because SELECTs are very common, it executes most commands at lightning speed (especially if you can fit it into memory). The 'De-normalisation' of this data sped up the SELECT queries 100 fold but only quadrupled the amount of memory that the DB required.

Sit down and analyse the types of queries that are common and then think about how you can make them work faster. There's plenty of help out there if you identify where the problem stems from.

[Frederic1]

I made a cache programme for my site, but then my server was storing too much cache files in the cache directories and file numbers crossed the limits and I ran out of Server Node.. then i deleted cache system.

I think you should create a cron task to delete the files every 24h or 12h so they will never reach your server limit. Also don't forget to cache *only* The Frequently Requested Contents, don't try to record every single file.

Trust me Tabish, nothing beats file caching for performance :-)

[edited by: Frederic1 at 4:11 am (utc) on April 28, 2007]