Reports of Google Dropping Pages Out of Index... possibly a bug?

"Main search results now fully fixed," per Google, but new problems in GSC Reporting.

[Robert Charlton]

Just a note that Barry, on seroundtable, is citing widespread reports of Google dropping pages out of the search index; and though there's no official word yet from Google, it appears to be a bug....
[webmasterworld.com...]

Google Is Dropping Pages Out Of The Search Index; A Bug?
Apr 5, 2019 • 7:48 am
by Barry Schwartz
[seroundtable.com...]

Something weird is going on with the Google search index. I am seeing wide spread reports of SEOs and webmasters complaining that their URLs and web pages are being removed from the Google Index. Lots of folks are noticing that a nice percentage of their web sites are no longer coming up for a site command or showing in Search Console as being indexed.

Sources include our April Google Updates and SERPs thread [webmasterworld.com...]

[aristotle]

This may not be the same issue, but for a long time I've occasionally noticed pages on my websites TEMPORARILY disappear from the search results for a few days. It's always just one or two pages. I've always thought that these temporarily removals might be some kind of glitch or bug. Like I said, it's been happening for years but only occasionally.

But these new recent reports suggest a systematic large scale removal. Is it a bug, or an intentional reduction of the overall size of the search index?

[Wilburforce]

It doesn't seem to be affecting my site on checking, but I don't check all that often, so pages may have gone and returned while I wasn't looking.

Has anyone tried to find out whether this is systemic (or whether it is confined to a single data centre or cluster)?

[Steven29]

@aristotle I have been seeing the same thing for the past year or two and tried to reach out to google to show them what was happening to no avail.

There is also a bug with Google that shows the "When found time".

I believe people are getting their articles into google before the content they are scraping by using old links google has discovered previously and 301 redirects. This makes it appear as if the person copying content had the content first. Try to tell anybody and they will say "That's impossible you can't change your when found time to google".

I'm sure everybody at one point has added an article to google it and it says found "x days ago" and you think hmm.

[jost]

Ive had many websites crushed by google over the years. Which is why I started moving away from being a professional webmaster. Just cant live in a world that google, the super monopoly, can just pull the rug at any moment.

But today, to my shock, my local business is gone.
Its the highest rated in its category on google reviews. Its verified on GoogleMyBusiness for years. 200+ perfect 5 star reviews. All glowing.
My brick and mortar. Been #1 on google in my niche for 6 years.
Now im zero. zeroooooo!

This better be a bug they fix quickly. This is insane. I got out of the web business because google destroyed me so many times. Now they are going to destroy my local business too. Hate this reckless company so much.

[aristotle]

jost -- I sure hope that what happened to your site is an error on google's part and that it will be quickly corrected.

Can you tell us how many pages disappeared? Also, Have you searched for unique Snippets of text (enclosed in quotation marks) from some of your pages to be certain that they're gone. Also, have you checked in GSC for a manual penalty?

[JS_Harris]

I've had entire pages suddenly disapear from search, as in get zero traffic for any terms, but this rarely lasts more than a day or two and happens very infrequently. If it's a blackhat tactic by someone else it's hardly worth the effort, I've yet to lose a page completely for longer than that.

[jrpfhc]

You know how If you google something that you rank for it’ll tell you your average in that position, clicks, and impressions?

I’ll see that report, it’ll say I’ve average the 1.7 position in the last 28 days, and..... the site is nowhere to be found. I am in the top 10 for over 1000 queries. Spot checking them and it appears to affect 10-15%. Just gone.

If it’s happening to others I’m not going to panic.

[tangor]

If it’s happening to others I’m not going to panic.

Then again, if it is happening to others, it might be time to panic. Because it might become permanent. The black box is opaque. Nobody knows what it really does.

[Shepherd]

We've not seen any of the reported issues. I have seen that some people are reporting that it may be and issue with Cloudflare and googlebot not playing well together. We do not use Cloudflare.

[jrpfhc]

Shepherd, that’s interesting. We definitely use Cloudflare.

[Jori]

I don't.

[SnowMan68]

I don’t

[Jez123]

I used google's URL inspection tool in Search Console and requested indexing of the page that had gone from the index. It reappeared in about 10 minutes and all was well but then it disappeared overnight again! I can't believe that if it's an intentional culling that I would be able to successfully request the page's return. I am hopeful anyway.

[sk7411]

Google is weird since the last update , while john did acknowledge about this issue a while back on a twitter reply , i am seeing another issue where url as root hostnames are shown in the mobile serps . I also understand that google selects the urls and might rearrange them as per their algorithm ,however showing naked hostname as URLs might seriously affect CTR . I am pretty much sure things were not like this until last week. Earlier the urls used to have meaningful breadcrumbs structure or atleast truncated with a nice hellip ... See attached image link for clarity .

[Rlilly]

Some of my pages disappeared- now showing again.. its a bug in my opinion

[azlinda]

I don't know whether this has anything to do with our problems or not.

I did a search for one of my unique pages on Google just now. I can't believe what I found! Instead of the URL being https://example.com/xxxxxxxxx, it was listed as
https://mail.example.com/folder/article. When I clicked on it, I got this notice: Did Not Connect: Potential Security Issue.

What is going on?

I performed the same search on Yahoo, and the correct URL came up.

[edited by: Robert_Charlton at 4:09 am (utc) on Apr 10, 2019]
[edit reason] Delinked "domain" urls and changed to example.com [/edit]

[broccoli]

I reindexed some missing pages in GSC yesterday - today they have gone again along with some others. Quite a few of my rankings have risen, but I can't figure out if that's because Google has deindexed some pages from a rival site.

[engine]

According to John Mueller, the issue has been fixed.

Sorry -- We had a technical issue on our side for a while there -- this should be resolved in the meantime, and the affected URLs reprocessed. It's good to see that the Inspect URL tool is also useful for these kinds of cases though!

[mobile.twitter.com...]

[v008370]

@Steven29 how can I see the When Found Time you mention? I'd love to see that data.

[Selen]

I noticed it a few weeks ago. Now Google doesn't show more than 30-35 pages indexed on any term. Try to type some very popular keywords like: "net", "money", "travel", or even the word "and" and you'll see there are only up to 30+ pages indexed. Same with the "site:" operator.

I've just checked: typed the word: net and went up to 27 page with the result:

Page 27 of about 268 results (0.91 seconds)

So it appears there are only 268 indexed results with the word "net".

[thepointer]

This issue is not fixed. I guess I'm on the second wave as I was not affected until John said it was "fixed". The posts that are being removed seemed random but resubmitting them in Search Console does restore them in about 10 minutes.

[swright]

This issue is not fixed. I guess I'm on the second wave as I was not affected until John said it was "fixed". The posts that are being removed seemed random but resubmitting them in Search Console does restore them in about 10 minutes.

Judging by what others have shared, re-submission only fixes the issue for an hour or so. The problem is we only have 10 submissions a day in GSC. One of the biggest screwups I've seen with G.

[thepointer]

I've resubmitted my first post almost 2 hours ago and it seems to be holding. It's just a bummer to find the random posts that are not ranking anymore.

Whatever they did to fix the first round of people trigger a second round of people to get hit. So messed up!

[Jori]

I never saw something like that. I think I have about 1/4 of my pages out of the index. Those pages that remains are seeing huge improvments though. Is it because my website is good, or it is because my competition is beeind also out of index ?

Google, what are you doing?

[swright]

I never saw something like that. I think I have about 1/4 of my pages out of the index. Those pages that remains are seeing huge improvments though. Is it because my website is good, or it is because my competition is beeind also out of index ?

Perhaps it's because your competitors also have pages out of the index. I'm seeing the same thing with some of my pages that are still indexed. One of them moved up a couple of spots today in the span of several hours and the pages that were overtaken are nowhere to be seen. I just tried to find one of them on the site of my competitor (using the 'site' command) and it's not showing up there at all so it's deindexed.

[tangor]

NOTHING is deindexed ... but the FILTERS have been shifted, for whatever reason (black box, we never know).

Don't ever believe your URLs have been "lost" or "removed". G (or any other SE worth its salt) ever forgets a url it has met.

Something else is going on, kiddies.

[Jori]

it's just semantics. If it's not in the index when I do a search, it's deinxed. But yes, I agree, G never forget a URL.
I think the best I have to do now, it's not to check my computer for one or two days.

[swright]

NOTHING is deindexed ... but the FILTERS have been shifted, for whatever reason (black box, we never know).

Don't ever believe your URLs have been "lost" or "removed". G (or any other SE worth its salt) ever forgets a url it has met.

Whether we call it deindexing or pages not showing in the index or something else, the fact is that obviously something is very wrong. Sure, Google has these pages somewhere on their servers, but has stopped showing them in the index for some reason and in the search results. I mentioned on the other thread how one of my pages disappeared today and some other page was ranking #1 for a query I was ranking. I looked for my page on my site using Google, and nothing was showing up no matter what I tried. So I went to GSC, fetched it, and Google said that it's in the index, but it still wasn't showing when searching with 'site:'. So I requested indexing again. Within a minute I could find the page on my site again and it was again #1 for the query, and the site that was ranking for the query was a couple of spots down. Many other people have reported the same - page disappears from the SERPs and the index, and the moment they resubmit it in GSC, it starts showing up in the SERPs in its previous position. Whatever it is, this is not normal behavior.

[Robert Charlton]

(((Not quite RESOLVED yet... It was a bug!)))

John Mueller reports that it was a "technical issue" that has been resolved, and that the affected urls are being reprocessed. In the Twitter thread where he comments, he's what I'd call necessarily evasive about what the issue was...

[twitter.com...]

Sorry -- We had a technical issue on our
side for a while there -- this should be
resolved in the meantime, and the affected
URLs reprocessed. It's good to see that the
Inspect URL tool is also useful for these
kinds of cases though!

Mod's note: Edited as Google changes its prediction.

[edited by: Robert_Charlton at 9:01 am (utc) on Apr 8, 2019]