Welcome to WebmasterWorld Guest from 34.204.189.171

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

How many seconds googlebot can wait for a page to load?

What about long SQL queries?

     
3:14 pm on Apr 24, 2007 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2005
posts:324
votes: 0


Hello All,

I wonder that how many seconds googlebot wait for a page to load? Suppose one of my dynamic page takes 20 seconds to execute a Mysql query, then will googlebot wait for that page?

Regards
Tabish

5:26 pm on Apr 24, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:June 7, 2005
posts:110
votes: 0


Googlebot dynamically adjusts its crawl rate based on the response time of your site. In most cases it will never request the next page until the previous request is complete.

20 seconds? Dude, you need an index on that table!

5:41 pm on Apr 24, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 9, 2004
posts:1435
votes: 0


We found out a while ago (due to an error which was quickly fixed) that Googlebot will wait more than 30 seconds for a page to load, and adjusts its crawl rate based on response times.

Similarly, if you have loads of pages, importance and serve pages in under a second Google will happily eat 80,000+ pages a day ;o)

5:57 pm on Apr 24, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 5, 2006
posts:2095
votes: 2


I agree with inbound, we had an error as well that was taking about 30 second and googlebot still indexed it and had a 200 code to it. We have fixed it, but I can attest to 30 seconds as well.
8:50 pm on Apr 24, 2007 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5848
votes: 198


Googlebot may wait 20 seconds for a page to load, but the more important question you should be asking is if your USERS will wait that long.

Secondly, you should be trying to figure out how to do things to get a much faster query. That is WAY too long for a public page.

10:17 am on Apr 27, 2007 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2005
posts: 324
votes: 0


My site is completely Database driven site and thousands of pages. Sometimes when i get much hits, my sql quaries takes much time to execute. I have optimized everything and made it As simple as possible but so many of my pages still takes 3 to 4 seconds to generate.

Shoudl i ask for my server upgrade, like more RAM or something like that?

3:22 pm on Apr 27, 2007 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5848
votes: 198


Are you on shared hosting or a dedicated server? If shared, move to dedicated ASAP- it sounds like other users are slowing your site down.

BTW, 3-4 seconds is a lot different from the 20+ seconds mentioned in your original post; 3-4 seconds is not all that bad. Chances are you can always find more things to optimize. Are your tables indexed properly? Are all your tables fully normalized (if so, often you can optimize things by de-normalizing tables)? Does MYSQL have query caching?

3:45 pm on Apr 27, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:July 7, 2005
posts:98
votes: 0


Use a cache.

Check the following php functions :
ob_start()
ob_get_contents()
ob_end_flush()

4:55 pm on Apr 27, 2007 (gmt 0)

Full Member

10+ Year Member

joined:Aug 22, 2005
posts: 324
votes: 0


I have a Dedicaed Server, yes, MySql Query cache is there.

I made a cache programme for my site, but then my server was storing too much cache files in the cache directories and file numbers crossed the limits and I ran out of Server Node.. then i deleted cache system.

Bytheway, my site has good hits.

Any other idea?

Regards

5:16 pm on Apr 27, 2007 (gmt 0)

Moderator from US 

WebmasterWorld Administrator lifeinasia is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 10, 2005
posts:5848
votes: 198


I don't know about MYSQL's query caching system, but you don't need (or want) to cache every single query- just the most commonly called ones.

Looking into upgrading your server's memory and/or processing speed may also work- you'll need to do some investigation to see if either (or both) is causing a bottleneck currently. Splitting the DB server to its own server may also be needed- it all depends on how your server is handling the current load.

But I also made some other suggestions- look into further optimization before throwing money at hardware.

6:23 pm on Apr 27, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 9, 2004
posts:1435
votes: 0


Running well-optimised MySQL queries on a dedicated server should not cause pages to take 3 to 4 seconds. You need a load of optimised DB calls to be getting anywhere near that.

Are you sure that you have implimented indexes properly? It could also be that your joins are overly complicated. There are many reasons why MySQL is running slowly.

You may want to search for 'mysql performance blog' on Google, it'll give you a few pointers. It'll also give you inspiration to dig deeper into specific problems.

A lot can depend on how strictly you follow normalisation rules, sticking to them 100% is not always best. An example of this is a 170,000 page dynamic site that I've just completed DB optimisation on. I wrote a program to 'flatten' the data from a properly structured source DB every evening. The flattened DB is larger than the normalised DB but, because SELECTs are very common, it executes most commands at lightning speed (especially if you can fit it into memory). The 'De-normalisation' of this data sped up the SELECT queries 100 fold but only quadrupled the amount of memory that the DB required.

Sit down and analyse the types of queries that are common and then think about how you can make them work faster. There's plenty of help out there if you identify where the problem stems from.

3:59 am on Apr 28, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:July 7, 2005
posts:98
votes: 0


I made a cache programme for my site, but then my server was storing too much cache files in the cache directories and file numbers crossed the limits and I ran out of Server Node.. then i deleted cache system.

I think you should create a cron task to delete the files every 24h or 12h so they will never reach your server limit. Also don't forget to cache *only* The Frequently Requested Contents, don't try to record every single file.

Trust me Tabish, nothing beats file caching for performance :-)

[edited by: Frederic1 at 4:11 am (utc) on April 28, 2007]