522 errors in GWT - host or content delivery network?

[onlinesource]

I feel like 522 errors are a pretty big deal because it's basically Google telling me that they had issues crawling part of my website.

I also can't a straight answer as to what is causing these errors. Google says this is often a problem due to connecting with your server. OK, except I use CloudFlare. Now, CloudFlare says my host is likely blocking their ips and they need to white list them. My host claims that "We have already allowed cloudflare IPs in server and there is no blocks in server end.".

So, what is the problem? Is somebody obviously lying to me? All I know is that I operate about 15 websites and 8 which use CloudFlare are all having issues ranking. The sites that don't use CloudFlare rank. I don't think it necessarily is a CloudFlare issue because it could be a plugin or some outside force preventing the pages from being crawled.

Is the anyway for me to check this and see the problem myself?

[not2easy]

What do your logs say? Check error logs and access logs to see what Google sees.

[lucy24]

What kind of "errors"? Google uses this term to cover any and all types of problems, including ones that aren't errors at all and can safely be ignored. For example, every time they ask for a page and don't find it, that shows up in gwt as a "crawl error" -- even if the URL has never existed anywhere but in the search engine's fevered imagination.*

If your site is down for one second in the course of a week, you can rest assured the googlebot will come by asking for robots.txt during that one second.


* Although this is the google subforum, it isn't strictly a google behavior. In some areas, bing is even fonder of nonexistent URLs.

[netmeg]

Ok, I'm confused. These errors - are there 522 (qty) of 404 errors? Or are there an (so far unknown) qty of 522 server errors?

Having done some Googling, I suspect it's the latter.

How many errors does GWT report?

Is this WordPress? Do you have a cache plugin installed?

Are you running cheapo hosting (which I define as hosting under eight bucks a month) on one of a number of notoriously inexpensive but not-optimized-for-WP-no-matter-what-they-tell-you host providers?

[onlinesource]

@netmeg. I have both, but I am trying to address 522 errors. Google says they can't crawl specific pages. I can go to any of those specific pages and they look fine. That is what baffles me?

The number of errors changes. Right now, there are only 6. Some of these are old product tags that no longer exist, some are product tags with parameters like "?orderby=price" (which I don't understand at all, and two are actual product urls. Again, if the url was missing I could expect to see a 404 error. 522 is Google basically saying they can't access it. That makes no sense to me.

When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

As far my hosting, I use a dedicated server paying nearly $120 a month. As far as cache plugins, so I don't run any.

@lucy24, maybe Google is just crawling me the second I go down? Anyway to fix that? Obviously hosts aren't up 100% of the time (98% to 99% is good) but in a split second, that 1% could kill me, right? As far as I know any failed attempts for Google to crawl me is not good, correct?

[onlinesource]

Not sure if I'll be able to post this or not, but this specifically shows server errors. It's a GWT screenshot. I blurred out anything personally to my site/account etc.

[i58.tinypic.com ]

For the most part I show a steady stream of 1 error to 0 errors and then BAM... 6/27 I go from 0 to 22 errors overnight! So, is there anyway to monitor what may have gone wrong on 6/27 specifically?

[JD_Toims]

Some of these are old product tags that no longer exist, some are product tags with parameters like "?orderby=price"

--

When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

First, don't worry about the pages that don't exist or the parameters that Google tries to add to generate a response from your server. Those are just Google being Google and reporting things in a "scary way" in WMT.

---

Second, 522 isn't an error code according to any HTML/RFC spec. I've seen; it's specific to CloudFlare: [support.cloudflare.com...]

This status code is not specified in any RFCs, but is used by Cloudflare's reverse proxies to signal that a server connection timed out.

[en.wikipedia.org...]

When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

It could be Google hits CloudFlare's URLs referencing your site hard enough to have CloudFlare access limited to your server by your host, so since you now CloudFlare's IPs are allowed, I'd start by asking if they have a request rate limit that might "kick in" if GoogleBot [or anyone else] hits CloudFlare hard enough to get blocked dynamically -- Sometimes, "allowing access" means they say, "Don't block these IPs", but when those IPs hit a server too hard some other system, such as DDoS attack blocking, kicks in and they get blocked at the firewall anyway, so that's what I'd start asking questions about.

[onlinesource]

@Jd_tomis, I'll pass that along to CloudFlare. I originally asked them about the 522 errors the other day and they said to me:

-----------------------------------------------------------------------

An error 522 normally indicates we weren't able to establish a TCP connection with your origin, this can happen if your origin's network is overloaded or packets are being dropped by a firewall due to rate limiting. Because CloudFlare operates as a reverse proxy the IP address your server will see is one of a limited number of CloudFlare IPs. In that sense, many actual visitors may all come from the same IP address, which can cause firewalls or security software that is not appropriately whitelisting the CloudFlare IP ranges to block this traffic as it may see it as excessive or malicious. We publish a full list of our IP ranges here so that you can whitelist:

[cloudflare.com...]

I would make sure that your hosting provider do the following:

1) Confirm that the CloudFlare IP ranges listed in the URL above are fully whitelisted from any security software, firewall etc to ensure there is no rate limiting or blocking of our edge servers.
2) Confirm that generally your network is healthy and not overloaded

This should ensure that CloudFlare can consistently make a connection to your origin server to retrieve content and serve it to your visitors

CloudFlare Rep

-----------------------------------------------------------------------

So, I then asked my server about white listing the CloudFlare ip?

-----------------------------------------------------------------------

We have already allowed cloudflare IPs in server and there is no blocks in server end.

Server/Hosting Rep
=========================================================================

That is where we left off. I will see if cloudFlare can shed some light on what you said. Either the host or Cloudflare, IMO is blocking Googlebot which could be for any number of reasons... maybe they think it's an attack. It makes sense to me.

[JD_Toims]

... maybe they think it's an attack. It makes sense to me.

It makes sense to me too, so please, let us know what you find out so others who find this thread have more insight as to how to deal with a similar situation, because it's one of those things that I'm sure could be not only an "annoyance", but could also have a "negative impact" on rankings at some point in time, if not immediately.

[lucy24]

Ok, I'm confused.

Not half as confused as me, since I simply barged in with the wrong assumption :(

:: insert classic jdMorgan wisecrack here ::

[onlinesource]

I heard back from Cloudflare.
////////////////////////////////////

Hi Clay,

A 522 is never a result of a rate-limit set by CloudFlare. Additionally, we have special rules in place to allow access for search engines to CloudFlare's network.

As my colleague [colleague's name] mentioned previously, a 522 is the result of CloudFlare being unable to establish a TCP connection to the origin server before the attempts timed out (15 seconds). This means that CloudFlare was unable to send the http request to the origin as no network connection to the origin server could be established.

Common reasons why this can occur aside from rate-limiting can include:
- A process on the webserver is consuming all of one or more resources at the origin.
- The domain is on a shared hosting platform where traffic or tasks for a separate domain on the same server is taking up one or more resources.
- Network connectivity issues between CloudFlare and the origin web server.

Please check with your hosting provider or server administrator to see if there were any reasons as to why the origin was unable to respond to all connections, or if there were any adverse network conditions. You may also want to check the origin's access and error logs for any corresponding entries.

[engineer's name]
Support Engineer

///////////////////////////////////////////////////
I sent this to my host. Their response?

-----

Hi,

We have double checked and see that your cloud flare IP's at our server end.

It has been shown already allowed cloud flare IP's and none of the issue in server end.

If you need any change in your cloud flare IP's , Please contact to cloud flare support team.

They will assist you in details for further.

Thanks for your understanding and co-operation with us.

Let us know, If you need any further assistance.

Best Regards,

System Engineer

------------------------------------------------

So right now, it's a lot of finger pointing and nothing is resolved.

mod's note, typically we don't allow email excerpt's as per ToS #9, but the responses seem boilerplate-like enough to keep

[edited by: brotherhood_of_LAN at 9:30 pm (utc) on Jul 21, 2014]
[edit reason] edited [specifics] out of email. [/edit]

[netmeg]

For me that would be a sign to change hosts and CDNs. But that's just me.

[onlinesource]

It's hard for me to believe that it is the host because I have other sites that are having no problems. I think it is the CDN. CloudFlare came recommend, could somebody recommend somebody better?