tracking bailout rates

What analytics are used to measure bailout rates on slow pages

[latimer]

We are looking for a good way to use performance analytics to determine impact of heavy spider bots activity against load times and bail out rates.

What do the experts use to measure the rate of bail out on pages?

Could Urchin's click path reporting be used to measure this?

Is there something similar that can be set up on AWstats?

I saw a post on google where someone suggested a 1 pixel gif at top and bottom of page to measure server process time. Another suggestion was to put the entire page in cold fusion tag to get a general idea of performance that could be checked with bot activity.

What methods are best for doing this?

Any thoughts on this appreciated.

[physics]

I'm not sure what to use to measure bailout rates. But you might consider serving bots cached pages to reduce the load and possibly solve the problem.

[latimer]

Thanks for that info physics.

We would like to do that but there is a bit of concern here that google may not like the redirects and we wouldn't want to end up with some sort of penalty. Any information that you could offer that would put us at ease on this issue would be great.

[physics]

I'm sure there are a ton of ways to do what you want in terms of measuring bail out rates as a function of spider activity. One way would be to count how many spider hits you have in your logs every hour of the day, so you have 24 numbers Ns. Then for each hour also track the conversion rate for non-spiders, C. Then plot the normalized C and Ns on the same graph or something and get an idea of their correlation (or more likely anti-correlation). You could get the page load times by using something like Cricket if you're using apache (search for page load times on goog), and compare the conversion numbers to that also.
But I think you'll find that if your page load times are longer than 2-4 seconds then you have a problem and such studies may be a waste of time anyway. In part this depends on how much people want to be on your site ... i'm willing to wait longer for a big matrix of flight info but not for 10 images of books.
If you decide to use a cache then yes you do have to be careful not to get in trouble with the search engines but it can be done. If you're using mod_rewrite (i.e. not redirecting google just serving the cached page when the dynamic one is requested) and the content served is almost identical I don't see how you're going to get caught and even if you did you're not really doing anything 'wrong'. The thing to watch out for is that if your cache program malfunctions your site could get nuked for a while. Test test test, and monitor.
Or if you don't want to use a cache then get more ram or a better server or even better do a round robin dns or whatever to spread the load. For this I guess you might have to prove it's worth the investment in which case you can use the spider load / conversion rate comparison plot.