Does page response time affect googlebot crawl?

[onlinesource]

I've read mixed things in the past about how a slow page ultimately effects the ability for Googlebot to properly crawl your site. Not sure how much of that is true?

I did notice last week in Google Webmaster Tools that almost 20+ urls of my sitemap were submitted but not indexed. I than ran a report on my sitemap.xml fiel using HEADMasterSEO software which showed that all of my urls had a 200 status code which was good, but about 20+ (the same number that were not indexed by googlebot) had a response time of 7.50 to 24.49 seconds. I don't know what a good response time is, but I assume that 24.49 is very high?

Now I have built my site using Magento and I notice that almost all of the "slow response times" are coming from category and product urls. Yet when I take one of the category urls that showed with a response time of 25.05 in the report and run that same url through bytecheck.com, I see that the "time to first byte" is 0.440. Does this tell me that the problem is not with the server necessarily or the host but with the page itself? Because 0.440 means that pretty quickly the server responds to the request but then the page itself takes forever to completely load? The report at bytecheck also shows a "speed download" is 5568.00 which is total seconds to load the page from start to finish, I believe.

Below is actually what bytecheck showed me with my url edited out.

FieldData
Url Effectivehttps://www.mysite.tld/category1.html
Http Code503
Connect Time000
Total Time0.442
Time Namelookup0.011
Time Connect0.013
Time Pretransfer0.031
Time Redirect0.000
Time To First Byte0.440
Size Download2459
Size Header470
Size Request99
Speed Download5568.000
Speed Upload0.000
Content Typetext/html
Num Connects1

I know these results will fluctuate as I request results at different times. In fact, I actually ran the same sitemap.xml through HEADMasterSEO multiple times and the response times for the exact same category url were 25.05 that first time, 23.60 the second time and the third time was 14.60. I think the shift downward might have something to do with the fact that I use a content delivery network which caches my site for faster loading times. Maybe the site is still caching and the more it caches, the faster it will load?

Still, the response time for products and categories is substantially higher than CMS pages. Is there a certain time that I should be aiming for? I'm not sure what role this plays in googlebot's ability to index or crawl certain pages but I do believe looking at the percentage of indexed vs. crawled urls in Google Webmaster Tool's sitemap report, that there is some truth to this? Although I don't think a product or category url will ever load as fast as a CMS page, I would like to get it to a point where Google indexes 90%+ of my urls if not 100%.

Like I said the numbers are much better on CMS pages but very bad on products and categories. I didn't know if there were other tools that would let me see the exact elements on the page that had trouble loading? Maybe I'm running a module that takes a lot of resources to load?

[Andy Langton]

Magento can have some appalling 'first byte' times, particularly without caching enabled (and even with it enabled, depending on your setup). If you can use a "full page cache" module you can make a substantial difference - I was pretty much forced to use an FPC system for a recent Magento site to avoid unpredictable >10 second load times. Of course, make sure you have the built-in caching enabled, and also try playing with compilation settings to see what works best for your site. Also an ideal to enable error logging and check out for any system errors or exceptions. And disable modules you don't use, and be choosy about the ones that you do.

As far as speed testing, there are a few established players, including [webpagetest.org...] [gtmetrix.com...] [developers.google.com...] and [yslow.org...] Avoid testing from your own machine.

As for what you're aiming for, first byte time (i.e. how long before your server sends back any response) needs to be less than one second. Better still, less than half a second, and ideally a fraction of a second. Users will not wait more than a few seconds before opting to try another site, and even if they visit, the possibility of random waits of 10 seconds is nowhere near acceptable. Aside from making the overall experience poor, it damages a user's sense of your site's reliability and trustworthiness - will they get a nervous 20 second wait after entering their credit card details?

[blend27]

RE: Http Code503

You have a BUG in Your code. 503 is Service Unavailable.

[onlinesource]

@Blender27. I just noticed the "Http Code 503" in the bytecheck.com report on that page. I just ran it again and it's 200. That is another issue I am having, likely due to the security settings but sometimes bots get a 503 request at random times. Not sure if it's too many requests at once by a bot so it's being blocked? Trying to get my host to resolve this.

[lucy24]

You have a BUG in Your code. 503 is Service Unavailable.

The 503 by itself doesn't necessarily mean anything is wrong. Unnerving, sure, but not necessarily lethal.

A server will return a 503 response if you've hit its quota for simultaneous requests (a number that is out of your power to change if you're on shared hosting). I see it regularly on one group of pages that call more than 100 (the server's current limit) supporting files. The same thing can happen if you're using a CMS that makes a great many server calls at the outset. Human users ordinarily won't notice it when non-page files are involved, because the browser waits a moment and makes a fresh request.

The only exception is if your excessive server calls are triggered by the page HTML itself. If this happens regularly, and you can't reduce the number of internal requests, you should look into moving to a server with a higher quota.

[onlinesource]

@Lucy24. I do have a dedicated server. I might need to ask the host about my settings?

I also installed the Google Tag Assistant for Chrome and ran a report on my site. The 503 error appears on several pages it tries to load. I read that it may have something to do with an outdated analytics code in my site? One of the errors I get it Missing ga.js script and I read that it might relate to changes to how the ga.js script is now added.

Trying to UPDATE my code.

Yet, the report from bytecheck.com has nothing to do with Googlebot so maybe my server does have limitations. I am going to speak with them.

[lucy24]

Start by looking at your error logs. For any non-403 error* they tend to be pretty precise about what went wrong. (That's assuming Apache. I don't know how IIS and, er, OtherThingy report the various errors.)


* I have never seen a 403 explained as anything but "denied by server configuration" which is maddening when I'm trying to figure out which aspect of the request triggered the [F].

[tangor]

What is the ad population on site pages? No one likes to hear this, but run no ads for a day or three and see what happens.

[onlinesource]

@tangor I have 2 ads on my site according to the ad blocker plugin I use. I don't see how to locate them because I'm curious what constitutes an ad to them? I can you there is a popup for the chat program and a popup about my site being secured. Unless there is hidden ad elsewhere? I don't see anything.

I also have another site that ranks very well and has 6+ ads according to ad blocker plus. That site ranks well. Won't say that getting rid of the ads won't help and I am going to look into ways to get this done.

[lucy24]

I have 2 ads on my site according to the ad blocker plugin I use.

Is this, or is it not, your own site that belongs to you and that you yourself administer? If so, you know how many ads it has got, and where they are located.

Now, if you have non-advertising content that an ad blocker is interpreting as ads (been there, done that) it becomes a slightly different question. But it's still an exceedingly easy one to answer. Since it is your own site, you know what content should be present on on any given page. Anything that is not visible is getting blocked.

[tangor]

Sounds like a lot of third party stuff (chat, popups, security, etc as well as ads) and that will, of course, SLOW down a page load. How much administrative control of this site do you have?

[Paddy Displays]

Sounds like a very common magento issue you are having ( slow category and product pages), which can be caused by many different things eg bad template design, bad server setup ( if you have a standard LAMP setup you can find that mod-php is whats slowing you down, php-fpm is much better. You might find you don't have a PHP Accelerator like APC or Xcache which can also help)

But this is really a technical megento question and you really would need a magento developer to take a look at it for you.