"Googlebot can't access your site"

[IanCP]

Ted told me I should be able to search for this topic discussed here recently. Unfortunately I can find nothing at all.

I received an email from Google with the subject "Googlebot can't access your site http://example.com/".

Which of course is rubbish. I rarely look at webmaster Tools but for quite some time now Google has had my site under two names.

http://www.example.com/ [correct] and;

http://example.com/ [incorrect] and the thrust of the email from them.

Over the last 24 hours, Googlebot encountered 1 errors while attempting to connect to your site http://example.com/. Your site's overall connection failure rate is 50.0%. You can see more details about these errors in Webmaster Tools

Remedy anyone?

Thanks

[1script]

No remedy but perhaps it will make you feel a little better: you are not alone! I'm getting the same notifications on a number of sites in my account and what they say is certainly not true: not only DNS was up all the time but also, why other sites, also served by the same DNS server, are not affected?

The funny part is: if 1 error is 50% of all attempts, then they only tried to load 2 pages from that site on that day? That's a bit too low for Googlebot, no matter how small and even how badly penalized the site it.

[g1smd]

You should register both www and non-www in WMT. Google gives separate crawl reports and crawl errors for each one.

It may have been a temporary glitch with the site, or may have be a bottleneck nearer to Google's end.

[IanCP]

You should register both www and non-www in WMT

The proper one has been there quite a long time. The non-www one Google did that all by themselves quite some time ago, hence the problem. I didn't put it into WMT. Strange.

Simply put: http://example.com/ is non-existent, so they can't crawl what doesn't exist.

[IanCP]

I'll await more answers but my gut feeling is simply to delete it.

[tedster]

What response does the server give when a regular browser asks for http://example.com/

[Sally Stitts]

Similar thing here.
In Webmaster Tools, under Health - Crawl Errors - Site Errors
All day long, I get -
DNS - Couldn't communicate with the DNS server
Server Connectivity - Request timed out or site is blocking Google, and
Robots.txt Fetch - Crawl postponed because Robots.txt was inaccessible.

I contacted my host, and they said there is no issue on our end.

All regular pages are working just fine.
Is Google messing up here?
.

[tedster]

Thw only thing I can think of is that somewhere on the web there's a link to example.com without the "www". Whatever the cause, it's certainly not going to cause a ranking problem if the hostname doesn't exist. You don't want it to rank anyway!

[indyank]

Yes I am hearing this from several people and also seeing this on a number of sites. Definitely looks like something wrong at their end as logs and monitoring services don't report any downtime.

[lucy24]

my gut feeling is simply to delete it

You mean delete the non-canonical domain from wmt? No. It will just come back. There's a place in gwt where you tell them your preferred name. Once they've assimilated it, future SERPs will eventually give only that form.

You also need to make sure there's a redirect in place from all wrong forms of your domain name to the one right form. That's in htaccess or config, not in gwt. There are, at a rough guess, eight thousand posts in the Apache forum telling how to do it. Or, if you're on shared hosting, you can cheat and let your host do it for you. They should have a checkbox somewhere.

[tedster]

Links to the "naked" domain name are relatively common on the web, I'd think. Registrars listing Whois info for domains they registered, for example. I think this one might haunt you until you take Lucy's advice.

[IanCP]

@tedster

What response does the server give when a regular browser asks for http://example.com/

Checking right now in FF.

Oops! Firefox could not find .example.com

From memory, ditto in IE.

@lucy24

You mean delete the non-canonical domain from wmt? No. It will just come back.

The very reason I said I'll await further replies for guidance.

[lucy24]

Oops! Firefox could not find .example.com

Oops indeed. Did you really type the url with a leading period, or did it just sneak into the post? Quick detour confirms that a leading . makes the difference between a variant name (example.com) and complete failure (.example.com doesn't exist).

Except that if I just type . and nothing more, the browser will helpfully supply the whole URL of this current thread.

[indyank]

The non-www one Google did that all by themselves quite some time ago, hence the problem. I didn't put it into WMT. Strange.

Simply put: http://example.com/ is non-existent, so they can't crawl what doesn't exist.

AFAIK that isn't the real issue. I have 301 redirected all other url forms to the correct ones and yet i see those google notifications for both the domain versions. Similar notifications were received by several other people I know.

[bsand715]

Yesterday I received one for a site I do not own or control. But several years ago, I did down load it on my cumputer to do some work on it for a friend. It is still in my C files, only place I can find that would assosiate the site to myself.

[morganstanly]

I think you have not used 301 redirect for your website. Used 301 Redirect and see your result anytime in any search engine see only one results with (www.). And add sitemap.xml and verify in webmaster tools.

[1script]

By the way: these errors don't only come to the non-www versions of domains. I've received a few for the proper www version. This probably means that that www domain is doing so poorly on Google that 1 visit is really-really 50% of the entire daily crawl budget. At least in my case it looks that way. So, I can actually believe that 1 visit could be 50% of the daily crawl budget for non-www domain if Google knows that this is NOT the canonical version.

Why do you need to crawl it then is beyond me. Perhaps for some technical reason that has nothing to do with ranking or indexing, perhaps some registrar-related business.

This made me think that the DNS error I've received on the proper www version probably means that for this domain Google has mixed up which is canonical and they probably think www is actually not the canonical version. Gotta go take a look!

[IanCP]

@lucy24

Did you really type the url with a leading period

Nope. I literally put in http://example.com/

@morganstanly

Used 301 Redirect and see your result anytime in any search engine see only one results with (www.). And add sitemap.xml and verify in webmaster tools.

That's an impossibility to do with http://example.com/ simply because it doesn't exist.

My take on this is that somewhere out there someone has place a link to http://example.com/blue-widegets.htm and that is the cause.

Similarly I've often had problems with Google and one or more dodgy links I tracked down to a blog somewhere:

http://www.example.com/green-widegets.htm[full stop or question mark]

Oh well.

[Andy Langton]

That's an impossibility to do with http://example.com/ simply because it doesn't exist.

By "doesn't exist" I assume you mean that there is no DNS entry. But it does exist because it's registered and you are using subdomains (www) of it.

Basically, the web (and Google) expect websites to work without a www. The message is actually a hint from Google that there is some value in creating the necessary DNS entry and redirecting without the www to the www - value enough that Google is crawling it, anyway. the simple fix is to create the redirect and then forget about it. if you don't want to do so, then as long as Google perceive value, you'll keep getting the error.

[IanCP]

Thanks Andy, I'll flick pass it to my support people to fathom it out as there is nothing I have access to in my control panel [as far as I know].

[lucy24]

Basically, the web (and Google) expect websites to work without a www. The message is actually a hint from Google that there is some value in creating the necessary DNS entry and redirecting without the www to the www

By exquisite coincidence, my most recent logs illustrate the point:

209.85.238.28 - - [dd/Sep/2012:aa:bb:58 -0700] "GET /google{hexadecimalstuff}.html HTTP/1.1" 301 541 "-" "Google-Site-Verification/1.0" 
209.85.238.101 - - [dd/Sep/2012:aa:bb:58 -0700] "GET /google{hexadecimalstuff}.html HTTP/1.1" 200 384 "-" "Google-Site-Verification/1.0"

That's what it wants to see: One redirect, one successful fetch. Both, of course, are the identical file :)

[phranque]

That's an impossibility to do with http://example.com/ simply because it doesn't exist.

at the bare minimum the virtual server at your example.com hostname should "exist enough" to accept a HTTP Request and respond with a sitewide 301 status code redirect to www.example.com which is your canonical hostname.

ideally the server hosting at example.com should be smart enough to redirect only to 200 OK results on www.example.com by avoid redirects to error pages or redirects to additional redirects.

[Robert Charlton]

That's an impossibility to do with http://example.com/ simply because it doesn't exist.

Something doesn't seem to be computing here (no pun intended).

IanCP, I'm going to guess that somewhere you're not understanding some of the technical vocabulary being used here, and that you may not be set up with software to analyze what's going on. Please forgive me if this isn't the case.

Bits of vocabulary can be a steep learning curve, I've found, because many of the people qualified to give you good answers naturally use language that contains terms you're not necessarily familiar with, and I can see where people here might be talking past each other. Let me suggest for now a few things...

- type http://example.com/ (with your domain instead of example.com) into the address bar of your browser and hit enter. What happens? Please report back to us, including the url "address" you see in the address bar.

- do the same with http://www.example.com/ and tell us what happens.

Now, I'm going to mention a specific tool here, something we don't usually do in this forum, but it's already been mentioned in our favorite seo tools thread in this forum....

Favorite SEO Tools
http://www.webmasterworld.com/google/4367011.htm [webmasterworld.com]

The tool is, among other things, a server header checker. It's URI Valet - [urivalet.com...]

Type http://example.com into the URI input field up top and click the Submit button.

Report back the requests and responses listed under "Server Headers Details" further down the page. They'll be in bold blue type.

If your preferred canonical was http://www.example.com, you would get see as the first item"

1. REQUESTING: http://example.com

...and if it properly redirected to http://www.example.com, you would see a blue heading further down the page saying...

SERVER RESPONSE: 301 Moved Permanently

etc... You'd get note that it's redirecting...

Redirecting to http://www.example.com

2. REQUESTING: http://www.example.com

...and then further down the page, you'd get a server response and a destination URI.

Try it also with http://www.example.com in the URI box. You should get at least a somewhat different set of responses.

Please report back on what all those are, and we can go from there.

[lucy24]

ideally the server hosting at example.com should be smart enough to redirect only to 200 OK results

You can do this on your own server, but I really don't think it's viable on shared hosting. The server would have to perform some equivalent of the -d and -f test on every request, and pore over your htaccess to make sure the request isn't coming from someone who will end up being blocked (core comes after all mods including rewrite). You're looking at a significant detour into a php script for every single request, because a server-level redirect on its own would happen before the request ever reaches your individual site.

You can cut back a little bit by doing the 301 yourself instead of letting the host do it. But it still won't affect your core-level lockouts.

But it really isn't that big an issue. I track redirects for other reasons, and the 301-to-403 sequence-- or 301-to-301-to-something-else-- is very rare. In fact the most common occurrence of 301-to-301 is from major search engines that intentionally ask for the wrong form of the name.

[IanCP]

@ Robert Charlton:

As advised, I checked http://example.com/

Response:

Server Headers Details
1. REQUESTING: http://example.com/
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; AIRF; GTB7.4; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 1.1.4322)
Host: example.com
Connection: Keep-Alive
SERVER RESPONSE: 301 Moved Permanently
Date: Sun, 23 Sep 2012 23:44:42 GMT
Server: Apache
Location: http://www.example.com/
Content-Length: 245
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset="iso-8859-1"
X-Pad: avoid browser bug
Redirecting to http://www.example.com/ ...

2. REQUESTING: http://www.example.com/
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; AIRF; GTB7.4; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 1.1.4322)
Host: www.example.com
Connection: Keep-Alive
SERVER RESPONSE: 200 OK
Date: Sun, 23 Sep 2012 23:44:42 GMT
Server: Apache
Accept-Ranges: bytes
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset="iso-8859-1"
Destination URI: http://www.example.com/

Then submitting http://www.example.com/ I get this response:

Requested: 2012/09/23 16:51:52
User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; AIRF; GTB7.4; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 1.1.4322)

Server Headers Details
1. REQUESTING: http://www.example.com/
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; AIRF; GTB7.4; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 1.1.4322)
Host: www.example.com
Connection: Keep-Alive
SERVER RESPONSE: 200 OK
Date: Sun, 23 Sep 2012 23:51:52 GMT
Server: Apache
Accept-Ranges: bytes
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset="iso-8859-1"
Destination URI: http://www.example.com/

Thanks very much for the heads-up on the SEO Tools.

[indyank]

You can see that the first one http://example.com/ is 301 redirected to http://www.example.com/. And that is what you want as your preferred url is http://www.example.com/.

When you requested http://www.example.com/, you are getting a 200 OK response and that is perfect.

So the issue isn't at your end at all.

[Robert Charlton]

IanCP - Looks like you're redirected just fine. I can't speak to the rest of your canonicalization, but the www subdomain certain appears to be handled properly.

It may have been a glitch, but I wouldn't lose any sleep over it. If you do a search on the email text and see that lots of others are receiving the same note, then it might be a glitch on Google's end, or else a classic bit of unclarity.

Maybe this is their new way of telling you that everything's OK. ;)

[kemcar]

I believe the www domain was not doing well with Google for the time being for that particular page. So I don't think there is anything you can do to solve it from your end.

[tedster]

My take on this is that somewhere out there someone has place a link to http://example.com/blue-widegets.htm

Exactly. For that very reason, you want to make sure that every URL that doesn't include a "www" gets redirected to the exact same file path, but using the "www" hostname. In other words, a redirect only for (or to) the root alone isn't really enough.

So I'd suggest you double check by taking a good internal "with-www" URL and requesting the "no-www" version. It should redirect as well.

[IanCP]

Thanks to all the good folks for both your generosity of spirit and your time on this issue.

In Australian terms?

"I luv's ya all"

Again thanks, especially for your patience on an issue way beyond my depth.

@ Robert Charlton

I owe you.

@Tedster

Thanks for your guidance.