indexed pages go up, ranking goes down

[goodbyedee]

GWT shows that the number of indexed pages nearly doubled and on 3/3/13 we had 190 indexed pages instead of 98 we had since then. The site used to rank top 3 places for many years for all the (very competitive) keywords it was targeting. After a month since the index update, rankings went down from 5 to 25 places depending on the keyword.

Could the new content that got indexed be the reason for the drop? Is there a way to find out which where the new-indexed pages?

[aakk9999]

@goodbyedee
If the number of indexed pages suddenly increases without you adding any new content, it could potentially indicate a duplicate content problem (the same page showing under two or more different URLs).

I think you need to do a bit of analysis on your site.

a) How many pages does your site have that you want /should be indexed?

b) Is there any reason why the number of indexed pages has doubled? I mean, have you added that much new content?

c) Have you done any other changes to the site around that period that could result in Google getting to pages previously blocked from Google?

d) If you look at GWT, do you see it reporting any duplicate titles/descriptions? This could usually be a good indicator of the same page being shown under two different URLs.

e) Since your site does not have that many pages, you could perhaps see all of 190 pages indexed on Google using the site:example.com search operator. I would use this to try to identify which pages have been added to index.

Of course it is entirely possible that the changes in ranking are not connected with the changes you are seeing in indexing. But I would first try to understand where the indexing changes came from and fix any issue (if there is) before looking further.

[goodbyedee]

Thanks for your detailed reply aakk9999,

I have checked all your suggestions and the only thing that could be of importance is that Google seems to have indexed pages that does not exist. These pages are from the blog and specifically indexes of specific tags that i have deleted many months ago. (e.g. http://www.mysite.com/blog/tag/tagname). Other than that, we have not added that much new content and there are no duplicate titles in GWT.

I will try to manually remove the tag pages from the index.
Thanks again...
.

[edited by: Robert_Charlton at 10:24 am (utc) on Jul 6, 2013]
[edit reason] disabled auto-link to make example link visible [/edit]

[aakk9999]

Google seems to have indexed pages that does not exist

This indicates to me that these pages are not returning 404 response code - which probably mean that you are not handling this area properly.

I would suggest that you check if the response code returned for these tag pages is 404/410 or 200 OK. If it is 200 OK then this is the area to fix first. I would also check a sample of a made up "non-existing" URL to see if the problem is wide-spread or only affects tag pages.

One way that you can check is to go to Google Webmaster Tools and use "Fetch as Googlebot" to fetch a tag page and also one non-existing (made up) URL and check what was fetched. It will show you response headers and you will be able to see if it is 404 or 200 OK.

<added> You can also use Live HttpHeaders Firefox extension to check response codes</added>

[goodbyedee]

The problem is not wide-spread but only affects tag pages..
The pages return 404 not found..

[Planet13]

If the pages return 404 not found then they SHOULD be ok.

[aakk9999]

If the pages return 404 not found then they SHOULD be ok.

I agree.

@goodbyedee

Google seems to have indexed pages that does not exist.

The increase of indexed pages to 190 - I presume you are confident this is because of these tags pages and not because of some other URLs that are indexed?

Also, when you say "indexed", what exactly do you mean? Do you actually see them in SERPs, either as a result of your keywords search, as a result of site: command or as a result of search of the unique text that used to exist on these pages?

If you do see them indexed, do you know if these pages have been re-crawled by Google since you removed them, and when (i.e. recently or long time ago)?

Do these pages appear in WMT under Crawl Errors, Pages not found section?

BTW, I am presuming you are not internally linking to these pages any more. I am also presuming you are not blocking these pages in robots.txt as if you are, Google will not be able to see 404 response.

[goodbyedee]

The increase of indexed pages to 190 - I presume you are confident this is because of these tags pages and not because of some other URLs that are indexed?

Yes I'm pretty sure about that...

Also, when you say "indexed", what exactly do you mean? Do you actually see them in SERPs, either as a result of your keywords search, as a result of site: command or as a result of search of the unique text that used to exist on these pages?

i see them in serps as a result of "site:" command. When i click on them i get the 404 page...

If you do see them indexed, do you know if these pages have been re-crawled by Google since you removed them, and when (i.e. recently or long time ago)?

these tags all belonged in a certain post. for various reasons this post has been deleted but the tags remained showing 0 results. After that i have also removed the tags but Google is still showing them...

Do these pages appear in WMT under Crawl Errors, Pages not found section?

Only one of these pages appear under "Not found"

BTW, I am presuming you are not internally linking to these pages any more. I am also presuming you are not blocking these pages in robots.txt as if you are, Google will not be able to see 404 response.

No internal linking and no robots.txt blocking

[aakk9999]

Considering the above answers, I would reckon that Google has not yet attempted to re-crawled these tag URLs to see http 404.

Also, when Google sees 404 for the first time, it may leave the page in index until the page has not returned 404 a few times on subsequent crawls - to make sure the page actually disappeared rather than 404 being returned as a result of some temporary error.

Returning 410 instead of 404 may speed up the process.

Also, watch your WMT - you should see these tag URLs appearing in Not Found section if Google has attempted to re-crawl these URLs.

[Pandacide]

@goodbyedee We have the same issue!

We deleted a full directory of pages, but a month later, we see that the index has not gone down. In fact, it went up. We checked and re-checked. Every single page throws a 410.

We have been reducing the directory size over the past 8 months and index count has responded nicely. It is rebelling this time over. No clue!