Google: 65 Search Quality Changes For August and September

[engine]

Our latest installment of search quality highlights is here with 65 changes to report for August and September. As you may recall from our last post, in cases where we don’t have a descriptive name, we are using a unique ID number. August and September were both busy months as we launched new features, expanded the Knowledge Graph globally in English, and worked towards building the search engine of the future.

Google: 65 Search Quality Changes For August and September [insidesearch.blogspot.co.uk]

I won't list them all, but there are a few that caught my eye.

[Robert Charlton]

That's quite a list. Here's a quick and very partial overview of some of those that jumped out at me which I feel are significant.

Note mention of "trusted sources" in several places, emphasis on ability to group and identify important phrases (onpage and in queries), and also emphasis on local context (and perhaps by extension, user intent)....

#82862. [project "Page Quality"] This launch helped you find more high-quality content from trusted sources.

#83197. [project "Autocomplete"] This launch introduced changes in the way we generate query predictions for Autocomplete.

LTS. [project "Other Ranking Components"] We improved our web ranking to determine what pages are relevant for queries containing locations.

#83443. [project "Knowledge Graph"] We added a lists and collections component to the Knowledge Graph.

#83442. [project "Snippets"] This change improved a signal we use to determine how relevant a possible result title actually is for the page.

#84394. [project "Page Quality"] This launch helped you find more high-quality content from trusted sources.

#84460. [project "Snippets"] This change helped to better identify important phrases on a given webpage.

#83901. [project "Synonyms"] This change improved the use of synonyms for search terms to more often return results that are relevant to the user's intention.

[kheadley]

One that I found quite interesting;

#82407. [project “Other Search Features”] For pages that we do not crawl because of robots.txt, we are usually unable to generate a snippet for users to preview what's on the page. This change added a replacement snippet that explains that there's no description available because of robots.txt.

Does this mean that Google still indexes pages that have been disallowed by the robots.txt perfect example search "terms and conditions" and find the result;

Terms and Conditions - The Times
www.timesplus.co.uk/welcome/tp_terms.htm
A description for this result is not available because of this site's robots.txt – learn more

This page has been blocked by robots.txt but is still indexed?

[londrum]

noindex doesnt mean that it cant be included in the index. it just means that they shouldn't crawl it
but google can get the link from other pages (that they are allowed to crawl).

i used to assume that noindex would take pages out of the index, because that's how most search engines do use it, and that's what we've all got used to, but apparently that is not the case

[g1smd]

This page has been blocked by robots.txt but is still indexed?

What do you mean by "indexed"?

Google records the fact that a URL exists as soon as it sees a link to it. It immediately adds the URL to its database, for later crawling.

A URL "exists" as soon as a link is created pointing to a web resource - even if it is subseqently found that the hostname doesn't respond, or there's no page by that name on that hostname, or that page crawling is blocked by a robots.txt rule. The URL itself still "exists" for all of the time that there's a link with that URL in, found somewhere on the web.

If the hostname responds but the resource is blocked by an entry in the robots.txt file Googlebot will not fetch it (but page preview might) but Google will still keep a note that the URL "exists".

In order to determine the HTTP status for the URL and index the content on the page, Googlebot has to fetch it and will only do so if it is not blocked by robots.txt.

The page might return 301, 404, 403 or other non-content status codes. If the page returns 200 OK, only then is the on-page content indexed. However, if the page itself contains a meta robots noindex directive, the content will not appear in any search results.

[Robert Charlton]

Does this mean that Google still indexes pages that have been disallowed by the robots.txt

g1smd's post covers it, but here's a long recent discussion which also discusses many nuances and almost all possible misinterpretations. We've discussed the topic here many times.

Pages are indexed even after blocking in robots.txt
http://www.webmasterworld.com/google/4490125.htm [webmasterworld.com]

Read the above thread, which is still open. You can post there if you have further questions, but please... let's not drive this thread, about algo updates, off topic by getting into robots.txt vs noindex once again.

Google has simply added the message noted in an attempt to clarify what's an oddly paradoxical sounding situation. It's not a search change... just an interface enhancement.


PS: londrum, you've got robots.txt vs noindex roughly backwards. Read the robots.txt thread cited above, which covers noindex as well. It's a confusing set of protocols with regard to search engine indexing.
.

[edited by: Robert_Charlton at 8:28 pm (utc) on Oct 8, 2012]

[Panthro]

Haven't read the list yet. I imagine Penguin is included?

[searchengineland.com...]

[superclown2]

#82862. [project "Page Quality"] This launch helped you find more high-quality content from trusted sources.

In other words, yet more brandspam ........

[ichthyous]

LTS. [project "Other Ranking Components"] We improved our web ranking to determine what pages are relevant for queries containing locations.

This caught my eye...I was on page one for several location based searches for my field and suddenly last wekeend I dropped to page three. All sorts of new sites have taken over the the first and second page, many of them not very high quality

[rzaweb]

What does this mean:
"and worked towards building the search engine of the future. "

Will they continue with the search engine?

[Robert Charlton]

rzaweb - Not to take this discussion too far off topic... it's worth noting that in the Search Quality Highlights blog post, the text "building the search engine of the future" is the anchor text for a link that goes to another blog post that describes several features of that search engine in more detail...

Building the search engine of the future, one baby step at a time
August 8, 2012
[googleblog.blogspot.com...]

That post suggests ongoing enhancement of the Knowledge Graph, as well as more context aware search... which among other things, means more emphasis on personalization and searcher intent.

Will they continue with the search engine?

If you mean is Google still going to rank web pages, yes... I think that the search engine will necessarily stay at Google's core, but interpretations of that may vary. Other types of search and knowledge retrieval are going to be added into the mix, particularly in mobile. See this discussion on that topic...

Google is committed to NOT being a Search Engine
http://www.webmasterworld.com/google/4504318.htm [webmasterworld.com]


Mod's note: Let's keep this thread on the topic of Google's Search Quality Updates for August and September, 2012.

[superclown2]

I believe it's all very simple really and that what is at the heart of the problem is Google's wish to give precedence to trusted sites. On the face of it this is an excellent idea but the people who run a lot of these trusted sites have rubbed their hands together and put out pages targeting just about every key phrase they can think of for every product that is even remotely relevant to their key products. The result is the awful pap we are now served with, whilst real information is buried under brandspam and this update has merely increased this.
Now they need to have another algo to tackle brandspam. What will they call this? Puffin? Parrot? Pig?