Forum Moderators: Robert Charlton & goodroi
per GWT Diagnostics - Crawl Stats:
848 average Time spend downloading a page (in milliseconds)
also topic from previous post [webmasterworld.com...]
per GWT Settings - Crawl rate:
although we "let google determine my crawl rate", when clicking the "Set custom crawl rate" the numbers change assuming due to google's assessment of how fast our site can handle it's bots. Variations over the past several months:
0.2 requests per second - 5 seconds between requests
0.1 requests per second - 10 seconds between requests
0.05 requests per second - 20 seconds between requests
0.03 requests per second - 33.333 seconds between requests
We are making changes to improve site performance, and wondering what numbers we should be shooting for and what others consider optimal?
Can we expect to see increased sales if nothing else were to change except the speed based on the stats above?
That said, these two numbers are really measuring different things. Googlebot's crawling numbers are not nearly comparable to the average user's load time. And the crawl frequency juss doesn;t enter into a site speed consideration at all - because that's set by Google and doesn't affect your load times one bit.
URLs displayed in the Example pages are collected from Google Toolbar... Data may also not be available for your site if not enough users (with Google Toolbar and the PageRank feature turned on) have visited your site's pages during the time period displayed.[google.com...]
gzip - compress your pages (ours are, but I think Google is aware of that bug).
DNS - we've got 3 DNS lookups because we run Adsense, Analytics and Addthis.
On a fast connection (mine for example), the pages load very quickly.
To solve the DNS problem, it wants me to serve all javascript locally. But will that really make a difference to endusers as far as speed goes?
I've removed every other js, like quantcast, etc. Instead of AddThis you can use your own share links.
And about DNS lookups; if the latency is low, the amount of lookups/files/images doesn't matter. If you have a global visitor profile, you should consider an anycast DNS provider.
Understood Tedster, but might there be a connection between the crawl rate setting that google has set, that cann fluctuate as seen with my numbers, and the site performance graphed speed? I compared the dates on the crawl rates listed above to the Site Performance graph and can't say for sure because each day isn't graphed, but looks like the the 33 secs between requests that was seen on 12-23-09 matches with the beginning of what is the largest spike in time on the Site Performance graph. Will be interesting to keep on an eye on this, especially as we make improvements.
"Does anyone have a fast site? We're in the 60% range right now."
BillyS - yesterday I saw an example posted on a blog that showed a site within the fast 20%. So yes, there are fast sites out there. And assuming that if those sites are allowing google to set crawl rate and then they check the box on the crawl rate page to see what rate google is set at, that the numbers would be much lower and without fluctuations compared to the ones I listed above.
Still curious to hear from others.
I've spent a *lot* of time with the Firefox plugin, doing every little thing on their list. But I've got a 4 month old site, and all this work doesn't seem to be doing me a lick of good. I've got 62 links in GWT, and about 15% of my sitemap is indexed, despite repeated crawling.
You can follow the guides, use firebug to optimize etc., in the end It depends on your visitors location (latency) and their connection speed.
That is an excellent point. I have read that in other countries, the internet connection speeds that people use to access the internet are not as fast as they are in the US, so if you are getting traffic from other nations, the internet connection speed combined with the latency factor are going to increase load time.
I was thinking that maybe I'm caught in some type of Caffiene related vortex? I dunno, the toolbar says my HP is 3/10 which I thought is not too bad for a newish site?
Any tips would be welcome...
CRAWLING
Server speed is a factor that helps for a better crawl, potentially more frequent per page, as well as deeper into the site. But it's not a factor for staying in the index.
INDEXING
The number of pages that actually stay in the index, rather than just get spidered, is related very strongly overall to the backlink strength of both the home page and how that link juice is distributed internally. Backlinks to deeper pages are also a big help. Many webmasters have noticed that there seems to be a formula (admittedly a moving target of a formula) for how many pages will be retained in the main index for any site. It's not a simplistic formula, however. It's not a flat percentage, nor is it a certain hard number.
RANKING
Page load speed is still not an active factor in ranking (or crawling or indexing) but it is definitely on the horizon as a ranking factor, possibly this year. The early notices and the push from Google to inform webmasters and give them somne tools and education began last year.
I'll wait for more consistency in the reports before I try to act on them :(
#1
On average, pages in your site take 4.5 seconds to load (updated on Jan 21, 2010). This is slower than 69% of sites.
#2
On average, pages in your site take 0.7 seconds to load (updated on Jan 21, 2010). This is faster than 94% of sites.
Both are database driven. Yes, site #1 is more complex, however, on all machines I test, they both load at about the same rate for me. I have gzip enabled, I have no "DNS issues" on either site. Go figure.
And for site #1, 1 of the reported example pages is blocked in robots, another no longer exists, and 1 is my home page, which is the fastest page of the site, and it is reported at 5.7 seconds, which is simply impossible, given my average is 4.5 seconds. 5.7 is impossible. Think about it. ; ) If it has found my home page to be 5.7 seconds, then all of the other pages of mine MUST be greater in time to download, which would mean my average is greater. So, the numbers are wrong. They may be reflective, but they are wrong.
So, I am nervous about google starting to use these stats because... why else would they start reporting them?, but at the same time, I know for a certainty the stats are BS. They really do not add up. They are impossible numbers. The kicker is "These estimates are of medium accuracy (between 100 and 1000 data points).", which is a disclaimer that was added this month. I daresay it should read "These estimates are of low accuracy (between 1000 and 5000 data points)."
So, my advice would be, try to implement the fixes they indicate, keep an eye on the reported stats for big changes as an indicator of a problem somewhere, but otherwise dom't obsess over them until the numbers start to get more accurate. Certainly, google must know these numbers are not accurate, and thus, not useful for mainstream use. I think it is revealing this tool is still in "Labs".
I suspect they have already accomplished much of what they really wanted to accomplish simply by placing this item in Labs: for sure this is helping reduce their robot load as many webmasters must have implemented gzip and other fixes... Beyond that, there are too many variables and too much inacuracy for me to really believe they can use it in SERP rankings.
For us, it's our loyalty to Google tools that are dragging us down. I'm thinking Google is smarter than - "this site is slow, let's knock it down in the SERPs." I'm hoping they're going to be providing us with more guidance on how to increase speed and not break any TOS.
"this site is slow, let's knock it down in the SERPs."
Google could use it as a tie-breaker.
Another problem with the template is this. WMT states we're pointing to the same information two different ways. For example:
sitname/template2009/images/widget.gif
sitname/template2010/images/widget.gif
We've turned off the old template, so the reference to the 2009 template must either be outdated information or from a cached result. The system is far from perfect right now.
Another problem with the template is this. WMT states we're pointing to the same information two different ways. For example:
example.com/?dir=directory&file=file.jpg
Did anyone see the interview with Marisa Mayer at Web 2.0 where she talked about a Google test which showed a half a second shaved from page load time resulted in a 20% increase in traffic and revenue? 20% might not be a lot to some, but for me it would pay for a new employee or two. You can complain that it isn't fair, or you can make it work for you.
per GWT Labs - Site Performance:
average 4.7 seconds to load Slower than 71% of sites.per GWT Diagnostics - Crawl Stats:
848 average Time spend downloading a page (in milliseconds)
You need to keep in mind that the Googlebot doesn't download images, and that accounts for a lot of download time.
P.S. Forogt to mention this: Site #1 is an international site, and site #2 is a local site. So this supports the thesis of user latency and proximity to the server. Again, suggesting that utilizing these stats for ranking may not be fair.
If your site is an international site, then it's going to compete in the SERP's against other international sites, right? Sounds fair to me.
By international site, I meant that it is getting users from countries where internet access is slower on average and who are far from me, vs. my local real estate site that gets only local visitors via comparably fast Canadian internet access. My point is that this may account for the difference in reported access times. The sites themselves are similiar in background architecture and server access.
And no, it is a fluke that many of my customers are international - I just happen to be willing to serve anyone anywhere - most of my competitors are US-based and US-focussed. Most in my sector don't serve intl customers, but I do. So, no, it isn't "fair" per se... For someone searching for product info, it wouldn't be fair that my product info ranks lower because my site serves a greater percentage of intl customers than someone else In fact, my ability to serve intl customers should give me more clout and authority, not less. It would be dumb to penalize me because I have a greater percentage of people who like my site who have bad internet access.
This is one of those off-page factors that have so many perspectives and variables that I don't think it can have a 1:1 affect on SERPs. I still believe the big picture here is google wants to lower googlebot load so it can crawl more, and it can crawl 10X as much by simply getting every webmaster to implement gzip and a few other items, and they are getting people to do this for them fast by scaring everyone into thinking that this may become a ranking factor soon.
Anyway, I just wanted to share my stats because someone above asked. I thought my stats would reveal the stats are inaccurate. I didn't mean to get drawn into a debate on the pros and cons of this as a SERP variable... sorry! My recent experience (proof) with google stats of every kind, including site performance, link:, site: and on and on, is that they are all inaccurate and incredibly misleading (probably deliberately so we canot reverse engineer the SERPs). Do not beleive any of them, and for god's sake, do not make any decisions based on any of them.
I do think it is wise to consider that page load time has been a ranking factor for years with Google. At Pubcon 2005/New Orleans I was told by two Google engineers that their best advice if I wanted to improve my rankings was to buy the fastest servers and fattest pipe I could afford. They said that Google is always trying to rate end user experience, and page load time is a huge factor in end user experience. Though it might be indirect, page load time would affect how many users Google would send to your site. I've heard similar advice at every Pubcon I've attended since, all the way up to Pubcon South last year.
I've done countless tests proving this ever since then. At one point I bought a new server every six months because the increase in speed would pay for the server quickly.
I would have to do some research to show it, but remember when Brett was having hosting problems and WW was down for a week or so, and then the site was moved to a new, faster host? (This was a few years ago.) I would bet that traffic increased at least 25% after the change. I wonder if Compete has records back that far?
For someone searching for product info, it wouldn't be fair that my product info ranks lower because my site serves a greater percentage of intl customers than someone else
Given Google's recent emphasis on 'personalised' results, I would expect that site load time is going to be a similar element - they will push a site up the results which loads (or should load) fast for that visitor, as measured by nearby people getting it to load quickly.
Reasonable?
And has anyone considered a sneaky possible reason for google "penalising" slow sites? They do offer web hosting, after all. :)
