Website’s pages inexplicably disappeared from search results

[monk_dzen]

Hi!

Some pages from the website I work for have completely disappeared from Google search results. It’s not that they are ranked low, or not as good as I think they should, it’s that they’re literally not there.

Take, for example, this page: <removed url shortener> This is a news article I wrote on <article subject>. Now, let’s search for the exact headline of this article: <SERP screenshot with 3 different sites having exact match title, some other results but no page in question>. What I see is my article, aggregated by other news sites. On the second page of the search results there is even a link to the tag page of my website, which contains the headline of the article: <screenshot of the last SERP result only going to a tag page on the site> But the original article itself is nowhere to be found.

I want to stress, that it is not that it ranks low, it is simply not there on any of the three pages of the search results. There are even a lot of links to some more or less irrelevant pages on my website which contain some parts of the article: <screenshot>

According to Webmaster tools, Google sees the article perfectly, it can be “fetched as google” and there are only 29 pages excluded from index by robots.txt There are no manual webspam actions against my website and no security issues with my site's content. What’s more, it is possible to find the article via site search <screenshot of SERPs> if I tell Google to look for it specifically on my website.

Since exactly the same thing happened to a significant number of other articles on my website, I’ve lost a lot of traffic, which apparently now goes to the people who aggregate my content.

I would really welcome any possible explanations to what happened or ideas on what to do to resolve this situation. Thank you in advance!

[edited by: aakk9999 at 11:49 am (utc) on Jul 19, 2014]
[edit reason] Exemplified, please no URLs or screenshots with site identifying details [/edit]

[aakk9999]

Hello monk_dzen and welcome to WebmasterWorld.

From what you have said here, your article is indexed (shows in site: search) but it seems to be filtered out from the results. It certainly appears you are a victim of scrapers.

A few questions: How old is the website? Have you lost the traffic on a particular date and if so, which date? Can you tie in this date with any known Google update?

Also, have you tried to search for your article by searching for a sentence in body text in quotes? If so, do you get your site? If you do not see it, then on occasions you will get "Repeat the search with excluded results included". If you click on this, do you get your article then?

[Planet13]

When I read monk_dzen's post, i get the impression it might possibly over optimization, and that it might be an external or internal linking problem.

Of course, it could be purely technical as well. A few years back I accidentally put a noindex tag in the homepage of one of my sites. Ooppsss...

[monk_dzen]

Hi aakk9999 and thank you! I really appreciate you help as I've read a lot of postings about similar problems, but haven't found anything that helps me.

To answer your questions: The website is 17 months old and we never had any problems with Google before. We’ve lost traffic gradually in a week or so starting on June, 8. After the Google Panda update on May, 20 we’ve actually got a significant boost in search traffic (it increased almost twofold and was stable until June, 8).

Sometimes I can find the article when I search for a sentence in body text in quotes and some time I do not. For example, I could find an article with a specific search text in quotes yesterday, but it is not there today. Using some other parts of the text in quotes the article can be found in ‘excluded results’ but not always.

Could you please tell me more about these scrapers? Is there anything to do about it if we are indeed their victims?

[monk_dzen]

Hi Planet13!

We have a news website with lots of content and a dedicated audience, so we never did SEO optimization beyond the basic things like meta descriptions, sitemaps and titles. We never bought links or optimized pages for any specific keywords, as it is our position that user experience comes first.
I’ve thought about technical issues (noindex, robots.txt), but I haven’t found anything. Of course, in both cases, it can be that I’m missing something obvious.

[not2easy]

Hi monk_dzen, a group of AdSense users recently discussed the issues of dealing with scrapers at great length here: [webmasterworld.com...] Some of the information shared there can give you a starting point. Although the two problems are not at all the same, the causes of the problems have a similar root. (Warning before jumping in to that - don't panic and make drastic changes. slow and sure is better long term.)

[monk_dzen]

Hi not2easy and thanks for the information, I'll look into it.

Here is one more detail on the issue I noted. My website included to Google News. When I search for the recent entries by exact title or an exact text fragment, I can find them on the SERP in the Google News block and I can also find them with the news search for 1-2 days after the publication. But they, as well as the older entries, can not be found with the standard Google search.