Welcome to WebmasterWorld Guest from 54.225.2.178

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

522 errors in GWT - host or content delivery network?

     
10:05 pm on Jul 17, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


I feel like 522 errors are a pretty big deal because it's basically Google telling me that they had issues crawling part of my website.

I also can't a straight answer as to what is causing these errors. Google says this is often a problem due to connecting with your server. OK, except I use CloudFlare. Now, CloudFlare says my host is likely blocking their ips and they need to white list them. My host claims that "We have already allowed cloudflare IPs in server and there is no blocks in server end.".

So, what is the problem? Is somebody obviously lying to me? All I know is that I operate about 15 websites and 8 which use CloudFlare are all having issues ranking. The sites that don't use CloudFlare rank. I don't think it necessarily is a CloudFlare issue because it could be a plugin or some outside force preventing the pages from being crawled.

Is the anyway for me to check this and see the problem myself?
11:48 pm on July 17, 2014 (gmt 0)

Moderator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:2746
votes: 61


What do your logs say? Check error logs and access logs to see what Google sees.
12:12 am on July 18, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13201
votes: 345


What kind of "errors"? Google uses this term to cover any and all types of problems, including ones that aren't errors at all and can safely be ignored. For example, every time they ask for a page and don't find it, that shows up in gwt as a "crawl error" -- even if the URL has never existed anywhere but in the search engine's fevered imagination.*

If your site is down for one second in the course of a week, you can rest assured the googlebot will come by asking for robots.txt during that one second.


* Although this is the google subforum, it isn't strictly a google behavior. In some areas, bing is even fonder of nonexistent URLs.
12:34 am on July 18, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12876
votes: 190


Ok, I'm confused. These errors - are there 522 (qty) of 404 errors? Or are there an (so far unknown) qty of 522 server errors?

Having done some Googling, I suspect it's the latter.

How many errors does GWT report?

Is this WordPress? Do you have a cache plugin installed?

Are you running cheapo hosting (which I define as hosting under eight bucks a month) on one of a number of notoriously inexpensive but not-optimized-for-WP-no-matter-what-they-tell-you host providers?
3:12 am on July 18, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


@netmeg. I have both, but I am trying to address 522 errors. Google says they can't crawl specific pages. I can go to any of those specific pages and they look fine. That is what baffles me?

The number of errors changes. Right now, there are only 6. Some of these are old product tags that no longer exist, some are product tags with parameters like "?orderby=price" (which I don't understand at all, and two are actual product urls. Again, if the url was missing I could expect to see a 404 error. 522 is Google basically saying they can't access it. That makes no sense to me.

When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

As far my hosting, I use a dedicated server paying nearly $120 a month. As far as cache plugins, so I don't run any.

@lucy24, maybe Google is just crawling me the second I go down? Anyway to fix that? Obviously hosts aren't up 100% of the time (98% to 99% is good) but in a split second, that 1% could kill me, right? As far as I know any failed attempts for Google to crawl me is not good, correct?
3:41 am on July 18, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


Not sure if I'll be able to post this or not, but this specifically shows server errors. It's a GWT screenshot. I blurred out anything personally to my site/account etc.

[i58.tinypic.com ]

For the most part I show a steady stream of 1 error to 0 errors and then BAM... 6/27 I go from 0 to 22 errors overnight! So, is there anyway to monitor what may have gone wrong on 6/27 specifically?
3:43 am on July 18, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:July 19, 2013
posts:1097
votes: 0


Some of these are old product tags that no longer exist, some are product tags with parameters like "?orderby=price"

--

When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

First, don't worry about the pages that don't exist or the parameters that Google tries to add to generate a response from your server. Those are just Google being Google and reporting things in a "scary way" in WMT.



Second, 522 isn't an error code according to any HTML/RFC spec. I've seen; it's specific to CloudFlare: [support.cloudflare.com...]

This status code is not specified in any RFCs, but is used by Cloudflare's reverse proxies to signal that a server connection timed out.

[en.wikipedia.org...]
When I use "Fetch as Google", asking Googlebot to crawl one of the 522 pages again, it crawls it just fine.

It could be Google hits CloudFlare's URLs referencing your site hard enough to have CloudFlare access limited to your server by your host, so since you now CloudFlare's IPs are allowed, I'd start by asking if they have a request rate limit that might "kick in" if GoogleBot [or anyone else] hits CloudFlare hard enough to get blocked dynamically -- Sometimes, "allowing access" means they say, "Don't block these IPs", but when those IPs hit a server too hard some other system, such as DDoS attack blocking, kicks in and they get blocked at the firewall anyway, so that's what I'd start asking questions about.
4:05 am on July 18, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


@Jd_tomis, I'll pass that along to CloudFlare. I originally asked them about the 522 errors the other day and they said to me:

-----------------------------------------------------------------------

An error 522 normally indicates we weren't able to establish a TCP connection with your origin, this can happen if your origin's network is overloaded or packets are being dropped by a firewall due to rate limiting. Because CloudFlare operates as a reverse proxy the IP address your server will see is one of a limited number of CloudFlare IPs. In that sense, many actual visitors may all come from the same IP address, which can cause firewalls or security software that is not appropriately whitelisting the CloudFlare IP ranges to block this traffic as it may see it as excessive or malicious. We publish a full list of our IP ranges here so that you can whitelist:

[cloudflare.com...]

I would make sure that your hosting provider do the following:

1) Confirm that the CloudFlare IP ranges listed in the URL above are fully whitelisted from any security software, firewall etc to ensure there is no rate limiting or blocking of our edge servers.
2) Confirm that generally your network is healthy and not overloaded

This should ensure that CloudFlare can consistently make a connection to your origin server to retrieve content and serve it to your visitors

CloudFlare Rep

-----------------------------------------------------------------------

So, I then asked my server about white listing the CloudFlare ip?

-----------------------------------------------------------------------

We have already allowed cloudflare IPs in server and there is no blocks in server end.

Server/Hosting Rep
=========================================================================

That is where we left off. I will see if cloudFlare can shed some light on what you said. Either the host or Cloudflare, IMO is blocking Googlebot which could be for any number of reasons... maybe they think it's an attack. It makes sense to me.
4:30 am on July 18, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:July 19, 2013
posts:1097
votes: 0


... maybe they think it's an attack. It makes sense to me.

It makes sense to me too, so please, let us know what you find out so others who find this thread have more insight as to how to deal with a similar situation, because it's one of those things that I'm sure could be not only an "annoyance", but could also have a "negative impact" on rankings at some point in time, if not immediately.
4:47 am on July 18, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13201
votes: 345


Ok, I'm confused.

Not half as confused as me, since I simply barged in with the wrong assumption :(

:: insert classic jdMorgan wisecrack here ::
3:25 am on July 21, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


I heard back from Cloudflare.
////////////////////////////////////

Hi Clay,

A 522 is never a result of a rate-limit set by CloudFlare. Additionally, we have special rules in place to allow access for search engines to CloudFlare's network.

As my colleague [colleague's name] mentioned previously, a 522 is the result of CloudFlare being unable to establish a TCP connection to the origin server before the attempts timed out (15 seconds). This means that CloudFlare was unable to send the http request to the origin as no network connection to the origin server could be established.

Common reasons why this can occur aside from rate-limiting can include:
- A process on the webserver is consuming all of one or more resources at the origin.
- The domain is on a shared hosting platform where traffic or tasks for a separate domain on the same server is taking up one or more resources.
- Network connectivity issues between CloudFlare and the origin web server.

Please check with your hosting provider or server administrator to see if there were any reasons as to why the origin was unable to respond to all connections, or if there were any adverse network conditions. You may also want to check the origin's access and error logs for any corresponding entries.

[engineer's name]
Support Engineer

///////////////////////////////////////////////////
I sent this to my host. Their response?

-----

Hi,

We have double checked and see that your cloud flare IP's at our server end.

It has been shown already allowed cloud flare IP's and none of the issue in server end.

If you need any change in your cloud flare IP's , Please contact to cloud flare support team.

They will assist you in details for further.

Thanks for your understanding and co-operation with us.

Let us know, If you need any further assistance.

Best Regards,

System Engineer

------------------------------------------------

So right now, it's a lot of finger pointing and nothing is resolved.

mod's note, typically we don't allow email excerpt's as per ToS #9, but the responses seem boilerplate-like enough to keep

[edited by: brotherhood_of_LAN at 9:30 pm (utc) on Jul 21, 2014]
[edit reason] edited [specifics] out of email. [/edit]

12:23 pm on July 21, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member netmeg is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Mar 30, 2005
posts:12876
votes: 190


For me that would be a sign to change hosts and CDNs. But that's just me.
3:34 pm on July 21, 2014 (gmt 0)

Full Member

joined:May 11, 2012
posts: 266
votes: 3


It's hard for me to believe that it is the host because I have other sites that are having no problems. I think it is the CDN. CloudFlare came recommend, could somebody recommend somebody better?
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members