Indexing issue of list page

[amiromidi]

Hi all, I am new here and sorry if this post is duplicated, I couldn't find any similar issue on this forum. here is my problem:
We have a listing website, each category has its own page with all posts listed inside it and we wanted to have category page for certain key phrase, Previously we listed all posts inside the sitemap and google properly indexed all of them , but here is the catch:
for each category, google chose one of the posts as a canonical for the whole category and ignore the category page itself,
below are our actions to fix the issue:
1-we blocked all posts (which started with /v/..postname) in robots.txt ----result---> google still resist to index category pages
2- then, we add noindex tag to each posts pages and removed /v/ from robots ----result---> google crawled posts pages and discovered noindex tags, but still ...
3- then we validated "Discovered - currently not indexed" pages and for 2 months, they are in pending state.
4- then, we removed all of those indexed posts(404 response)

It's been 7 weeks, and every week, we tried to manually index those category pages, but still :
** "Page is not indexed: Discovered - currently not indexed"
** Referring page ---> /v/PLEWSXyL

good to know that, all of those category pages which started with noindex tag for their posts , all are indexed and work like a charm
also, I need to mention, we don't have quality content issues and no problem with crawl budget
the total number of pages are almost 35,000 , but at the moment we need 1000 of them to be indexed


Thanks for any help, hint...

[1][[b]edited by[/b]: amiromidi at 9:46 am (utc) on Sep 9, 2022][/1]

[Sgt_Kickaxe]

Just because Google isn't indexing a page doesn't mean they aren't passing link value through them.

That being said, Google using the url of an article as the canonical for the category page can be problematic. I've only seen this when a new category has 5-6 articles in it and each page also has a similar posts section with the same links, all of which show the same snippet... it becomes duplicate content too.

Try adding text to the category page to let users know what the page/topic is about and look at the similar posts section to see if maybe each page has too many of them. Make sure all of them are relevant and consider reducing how many are showing.

Lastly, you're making a lot of changes and 1 week is probably not long enough for any of them to fully bake into Google's rankings. Improve the content and category page if possible and let 2-3 months go by before making more changes... imo.

[amiromidi]

Thanks for your kind explanation, however, each category page has its distinct content with over 1000 words describing the key phrases for that particular category and there is no any similar internal links among them, also as I mentioned before, its been 7 weeks after our last action, and 4 months since we had encountered this issue.

[amiromidi]

It's worth to mention that, all of new category pages which we added after (having noindex tag for posts inside each category) were indexed and working as expected
The problem exists for previous category pages which previously had their posts indexed.

[not2easy]

It sounds like this is a WordPress site since you mention Categories and Posts. It is too easy to have duplicate content problems if you try to index posts and categories because the same content will be found under each of those (Categories and Posts).

You need to decide which URLs give your visitors the most useful landing pages. If it is Category pages, then you do not try to index posts separately. But disallowing crawl does not remove URLs from the index. The canonical link for each post would need to be set to its category and your posts sitemap removed. You will need to submit a category sitemap to get those Categories indexed.

Google is aware that WordPress offers many ways to show the same content. It is up to you to set the correct options and not try to index the same content under multiple URLs.

Preventing crawling of the /v/ URLs means that Google has not been able to see the new "no-index" tags you have added. That can confuse things and slow down the changes you hope to see. Also, as mentioned, a week (even a month) is not nearly enough time for Google to re-sort your indexed URLs. Oh, and they don't like it when the Verify URL tool is used for submitting to index multiple times. That will not speed it up.

[amiromidi]

Thanks not2easym here are some clarifications:
1- Our website is not based on wordpress, (React JS + Next JS for frontend), and we are monitoring accessing of each page with on-premises Sentry app. there is no duplicate issue at all.
2- As I've explained in my first thread, google has access to each post page now (we removed blocking of /v/... from robots) after action #2m and Google can see the noindex tags (this issue only exists for old category pages which their posts previously were indexed)

Are you suggesting to
1- removing noindex tag from each post page and set the canonical link of each post back to their category page
OR
2- keeping the noindex tag and just add canonical link of each post back to their category page?

[not2easy]

Sorry about the wrong guess there amiromidi.

Yes, I would use your #2 idea: keeping the noindex tag and just add canonical link of each post back to their category page.

[amiromidi]

Also to underscore the point, it's been 7 weeks since we removed (404 response) all of old posts under those category pages and replaced the with new post with completely different content and url, BUT, it seems google still memorized all of them as canonical and ignore the category pages

[not2easy]

Oh, Google will never forget an URL once they have tasted it. If you can control the response headers, a 410 (Gone) response can help. The 404 says "Not Found" so they keep trying.

[amiromidi]

Thanks again not2easy, We followed your guidance:
1- responding with 410 for deleted posts
2- add canonical tag back to category page for each post and keeping the noindex tag for them

waiting for result, fingers crossed

...any other recommendation?

[tangor]

Let us know what happened ... in December. TIME is what is needed at the moment.

[amiromidi]

For sure