Google Search Results Showing No Meta Descriptions

[mrengine]

I am seeing a lot of search results in Google that say:

A description for this result is not available because of this site's robots.txt

Failing to add a meta description, or blocking the page with Robots.txt, really defeats the purpose of search. Why does Google even list these pages when people can't even get an idea what the page is about? Is anyone other than me seeing a lot of these pages indexed at the top of the search results?

[phranque]

if google discovers a link to your site it may decide to index that url despite the fact you have excluded that url from being crawled.
it's not google's fault this meta description appears - that is due to your preference.

what you probably want is a meta robots noindex element for those pages and remove the robots.txt exclusion so googlebot can crawl those urls and see the noindex signal.

[Lame_Wolf]

Old news. Reported here over a year ago.

[JD_Toims]

Yup, there was actually a "wtf should we do" question wrt robots.txt exclusion by MC that allowed input "from the community" a few years ago -- After reading the reasoning behind why they include blocked URLs the suggestion of "just telling the truth" was made.

One of the reasons for the inclusion was "Google looks bad if they don't show a URL for a 'major site' that should be in the results, even if they don't want their URLs crawled." == "Bad Google user experience"

Another was for "inadvertently blocked pages where the site owner was unaware they had somehow blocked Googlebot from crawling the pages" == "Huge negative for the site owner"

So, the decision was to continue including blocked URLs where there were "enough links to warrant it" [enough being subjective and subject to change, of course], and then to try to "give a message" about why there was no info presented about the result.

Apparently, the "just tell people the truth about why there's no info" suggestion stuck...

[phranque]

this has been going on for about 15 months.
A description for this result is not available - Google SEO News and Discussion forum:
http://www.webmasterworld.com/google/4490309.htm [webmasterworld.com]

before this the description would either be blank or it would be constructed from the context of the linking document.
at the time i assumed the change was made for two reasons:

- many webmasters complained they had no control over the SERP content for their site when google decided to index their excluded url, so google gave the search result a generic flavor.
(i think the correct solution is to let it rank for the query but provide a descriptionless result and perhaps even url-only anchor text.)

- the generic non-description is an easy and universal way to pressure webmasters to allow crawling of excluded content.

[lucy24]

I'm a little confused by the thread title and first post. What does the meta description have to do with anything? Where the search engine says "A description is not available" it could just as well say "A snippet is not available".

It isn't specific to google. Bing does the same thing, with the wording
:: detour to check ::

We would like to show you a description here but the site won’t allow us.

duckduckgo and yahoo say the same thing.

[mrengine]

I'm a little confused by the thread title and first post.

Allow me to clear up the confusion. As we all know Google has included pages in the serps that are blocked by Robots.txt files for quite a while. This is not new. But ranking pages that are blocked by Robots.txt in the top 3 positions I find to be a bit odd. The reason why it is odd is that Google is giving good ranks to pages they have been blocked from viewing.

As I originally noted in my first post, is anyone other than me seeing a lot of these pages indexed at the top of the search results?

[netmeg]

Always been the case. Presumably there's enough off site signals and/or there are no competing pages with enough discernable authority to outrank them.

[mrengine]

Always been the case. Presumably there's enough off site signals and/or there are no competing pages with enough discernable authority to outrank them.

I've never seen these pages rank in position #1 before and above other reputable businesses in the same industry. So much for content being king...

[lucy24]

So much for content being king...

Different search engines have different standards. (Duh...)

When I want to do a quick check for handling of roboted-out pages, I search for yellow widgets and see if my test site (yellowwidgets.com) comes up. In bing, yahoo and duckduckgo it's #1 with the "we can't show" box; in google and yandex it's nowhere, though for different reasons. yandex apparently doesn't index roboted-out pages. google does-- but in this case it only turns up if you do a site: search.

So bing gives more weight to domain name alone.

Analogously, if I search for the exact name of a roboted-out directory, it comes up in bing-- eventually-- but not in google. Surprisingly, g uses exact spelling while b insists on throwing vague approximations into the mix.

[edited by: aakk9999 at 11:51 pm (utc) on Nov 14, 2013]
[edit reason] Replaced blue widgets with yellow widgets - blue widgets . com is a real website [/edit]

[Robert Charlton]

I've never seen these pages rank in position #1 before and above other reputable businesses in the same industry.

It completely depends on the query and other ranking factors. Yes, I saw this kind of result ranking #1 the first time I posted about the issue, back in 2003.

So much for content being king...

In the situation I described in 2003, it was about content quality... in combination with the authority of the url which was returned in the Google serps. The result was for a co-branded dupe of a client site (the dupe blocked by robots.txt) that was receiving a home-page link from a major newspaper chain. The "reference to the page"... ie, the url of the co-branded content... was ranking up top. I was upset that a copy of our material was outranking us (hadn't been told that the client was going to do this).

Are you observing this in a competitive search in your niche that affects you, or for that matter in any competitive search? I ask because I do a lot of searching, and it's not something I see very often. Generally, when I have seen it, it's on something like a web-based shopping cart that hasn't properly canonicalized the site, and is instead incorrectly using robots.txt to block the non-canonical urls that lead to the same content.

For a good reference albeit long read on the issues, take a look at this thread...

Pages are indexed even after blocking in robots.txt
http://www.webmasterworld.com/google/4490125.htm [webmasterworld.com]

[mrengine]

Are you observing this in a competitive search in your niche that affects you, or for that matter in any competitive search?

Really neither. I was just shopping/looking for a local company that both supplies and installs the widget.

Thanks for the link. I'll do some reading. Even though this has no impact on me directly, it does speak to the present state/quality of Google's search results - neither of which I find very impressive. Bing delivered a more precise and diversified set of results that did not include questionable pages that were blocked by Robots/txt.

[netmeg]

Recently had a situation where a client insists they have to have duplicate category pages on an ecommerce site. I fought long and hard (because there was no easy way to NOINDEX all but one each) but I lost, and they insisted blocking them with robots.txt was enough; the search engines were smart enough to rank the "right" one.

Well lo and behold, the search engines (Google, Bing and Yahoo) were NOT smart enough, and they ended up with the number one spot in all three engines with the appropriate block message in the snippet. Looked really stupid, too; it's a well known search phrase and they're the obvious authority source for it. After I pointed that out, I got NOINDEX programming.

In this case, the pages had already been evaluated by Google; they don't re-evaluate them after you block the crawl (because you told them not to, d'oh) and so OF COURSE a ranking page lost its snippet.

Stuff like this happens all the time.