Welcome to WebmasterWorld Guest from 54.80.188.87

Forum Moderators: Robert Charlton & goodroi

URLs for Certain KWs Getting Completely Omitted from SERPs

     
6:50 am on Oct 7, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


Hey everyone,

For the past 1.5 years, we've been noticing that some of our URLs are getting completely omitted from search results for particular KW variations. Looking over at Search console, they do appear to be indexed properly, and nothing holding them back. Additionally, the canonical is specified. If that URL has a pagination sequence (triggered as a result of user-generated content), along with sorting parameters, everything is set up in accordance to Google's documentation, as well as to the advice of professional thought leaders in the industry. For certain URLs, we rank just fine, and it fluctuates within reason, just as it does for anybody. But then for some reason many URLs just completely get omitted from results. Occasionally, they'll flicker back and forth, some stay, some go, and for some, it's a never-ending cycle of coming back then disappearing.

In certain instances when a particular URL gets omitted, it later returns to the SERPs under a different URL of the pagination sequence. For example, https://example.com/topic/ would go MIA but later return to rank much lower but under https://example.com/topic?page=5

At first, the issue wasn't too big, but over the past 1.5 I am beginning to notice more and more URLs are particular query variations just get omitted. After each major Google update, from which we weren't negatively affected, I notice this issue get “magnified” and spread. again, for the URLs that actually rank, they rank strongly, but some random ones get omitted. The code structure is identical all across the board and we can't seem to detect any particular pattern within the pages that are disappearing from the SERPs.

In fact, we've also noticed that when we publish new content they never appear in the SERPs, although everything indicates that they're properly indexed. Occasionally, they appear on the first page for a day or so, and then vanish again.

We've investigated over and over, and no one, including us, can seem to identify a root issue. Any help here would be very much appreciated as this is beginning to negatively affect us as this issue is spreading across more URLs. Thanks!
8:25 pm on Oct 7, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


Hi HereWeGo123,

Are other pages at your site placing for those searches? Are you competing against yourself?

About 2 years ago Google started omitting some pages from the index if they too closely matched the same metrics. That's why it is best to make each page unique and target a specific querry.

You can no longer just think in keywords. When designing pages for inclusion in the search index, it's now crucial to think in search querries. What question does your page answer? What is the intent of the user?
11:32 pm on Oct 7, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@keyplyr+ – Each URL is completely unique to its own subject matter. We are a review website, and I am referring to Kws unique to the product name + review (and reviews). Occasionally, although not too often, a URL won't appear as a result of the homepage of our site ranking for the same KW on like page 6 or 7 because the frontpage has the latest content added and mentions it by name, so it's hijacking it. But again, it's not too often. Most of the time, the URLs just completely get omitted. We have unique content, often authored by experts and/or expert first hand sources.
2:31 am on Oct 8, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


I salvaged a site for a client that believed his pages were unique,but some were not getting indexed. It came down to his architecture. 32% of the generated mark-up was the same from page to page, even though the actual content was different. Just a FYI.
6:42 am on Oct 8, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@keyplyr + – Thanks for that. When you reference “generated mark-up”, isn't that normal practice though (Unless I'm misunderstanding something)? All the code is the same across the board for content pages. Each page has a unique rating comprised of user reviews and uses the same infrastructural template (much like a WordPress blog).

Regarding Site Architecture, I can possibly see that as an issue too. Can you be a bit more specific on some of the issues you've noticed in your client's site architecture that raised red flags?

So many of your client's URLs returned back to the SERPs after being previously omitted?

I am also inclined to think that there has to be some unique technical issue that is causing this.

Thanks again!
7:26 am on Oct 8, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


I use Server Side Includes (SSI) to install sections of the top header & the footer section of all pages, as many do. That ratio of code, combined with other code, in relation to the unique page content is between18% to 22%. That's very good and because I stay aware of it when I create pages.

Some Content Management Systems (CMS) or Database (DB) driven architectures may significantly raise the code to content ratio, possibly to a level that is indicative to something included in the indexing algo of Search Engines and result in negative results.

That was the case for the client I mentioned above. I made additional changes so it's difficult to say exactly what I did to help his site, and likely a combination of things.

Now, as the SE algorithms evolve to better adhere to user intent by making use of AI learning, I feel it is required to keep up with those advances, not just fix perceived problems.

Nowadays, each page, as much as possible, needs to fulfill user intent by answering questions (since a large percentages of searches are phrased as questions) or specifically provide information that best suites the request. Search results are much more customized per user than they ever were before.

By their nature, some sites won't be able to adjust to the evolving indexing criteria. They once ranked well bringing in good traffic, but times changed and they have been left behind or are somewhere in that process. There may be little that can be done without a total restructure by a profession team.

Other sites can *keep up* fairly well by making small title and content adjustments. Knowing your audience and their what/why/how gives you insight to what possible changes could be done.
5:07 am on Oct 9, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@keyplyr – Thanks again for such a diligent and insightful response. Gives us a lot to think about and consider, of course.

One last question for now – Regarding architectural issues – We have Top parent categories and the products are associated with the appropriate category, obviously. About a year ago or so, we did a NoIndex,Follow on all parent category pages. Around that time is when we began noticing this issue spread of URLs getting omitted from the search. A thought leader said back in Dec 17 that if you NoIndex, Follow Google will still crawl them, just not index them (hence the name), but if it remains long enough, they will stop crawling them and it will have the same effect as a NoIndex,NoFollow.

Could the fact that the top parent category pages contain a NoIndex,Follow have something to do with this?
5:17 am on Oct 9, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


Certainly could. I don't know your reasoning for adding the NoIndex but I would remove it and other directives from all intersite links. Let Googlebot do its thing.
5:47 am on Oct 9, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


The reason we added a Noindex, Follow to the category pages is because the category pages did not contain any unique/compelling content or functionality so we wanted to reduce the ratio of “low quality” pages (category pages). The only value the category pages added to the user is just to browse products/organization. So the only benefit user gets from them is if they click on a category page, they can see all the associated products. We were advised to do so…
5:54 am on Oct 9, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


Well you could test a few pages and wait a few weeks to see what happens, but you said the dates were close enough for what we call "causality."
1:19 pm on Oct 9, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


If someone advised you to add a "Noindex,Follow" meta tag, maybe you should help them learn that "Follow" is not a directive to add to a link or a meta tag. The default is "follow", and you can use nofollow, but 'Follow' is not on the list of things you can add.

There's a good place to learn about the directives that can be used for Google bots here: [developers.google.com...] and it also explains when and why to use them.

11:53 pm on Oct 9, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy thanks! Yeah, so if we add a NoIndex, DoFollow (which I see is not a directive as you said), does it automatically disregard that 'Follow' and just follow links on that URL by default since we did NOT specify a NoFollow?
12:09 am on Oct 10, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


Why use a noindex for the category pages? There was news out about Google dropping crawls on noindex pages. I don't know the specifics, but they're in the forums here if you want to read about that. If they do stop crawling noindexed pages, how would they crawl the URLs linked from those pages?
12:27 am on Oct 10, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy – And I can see the logic now. That may be partially, if not fully related to the reason why many different KW variations for some URLs appear omitted from the SERPs (As I described at the beginning of this thread). We submit a new article through GSC, it appears for a bit and then vanishes. Yeah, I just camr across a few articles that Google eventually stops crawling if it sees a NoIndex, Period.
7:02 am on Oct 11, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy – So you would recommend turning off the NoIndex on category pages? The reason we turned it on initially was because those top parent category pages do not offer much “content value”. They are just there for organizational purposes to display all relevant product reviews/articles for that category as well as provides sorting products by criteria.
7:27 am on Oct 11, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


@HereWeGo123

If it was my site and I had the time and motivation, I would remove those category pages from the herichary. Make it simpler. Send visitors directly to the product page.

This could be a restructure of your site architecture - or you could just redirect via htaccess. I suggest the latter.

You could still group the product pages by category in your site navigation without actually using dead-weight category pages.

Eventuallly, those category pages will be removed from the search index. The product pages *should* aquire some additional link juice and you've removed any obstacle to their indexing
9:14 am on Oct 11, 2018 (gmt 0)

New User

joined:Oct 11, 2018
posts: 1
votes: 1


I also have few Wordpress blogs, and I am getting the same issue, i.e., index fluctuation. Sometimes, it shows, 350 pages index and some times, it shows, 200 pages indexed....... but it never showed less than 150 pages.......

What I am assuming is that the 150 pages have original content, otherwise other pages have some portion of content from other pages....

I think Google is able to identify how unique your page is compared to other pages of your website...
9:25 am on Oct 11, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12792
votes: 880


Hello wildweb and welcome to WebmasterWorld [webmasterworld.com]
6:29 am on Oct 12, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


@HereWeGo123
Yes, I'd get rid of the noindex tags, at least until you can find another way to let visitors and bots find those product pages. If there are not other means such as navigation to each page other than the category pages, I would want those crawled. "index" and "follow" are both defaults for Googlebot.
2:20 pm on Oct 12, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy – Thanks for that tip.

Just to clarify. Here's the current process: We add product review editorials content, associate them with the appropriate product category. And submit those new particular product review URLs to Google Index. For a few days or so, those most recent product reviews appear on our homepage as the most recent content added and then eventually get pushed off the homepage as new content is added. So after that, essentially, Google can't get to those pages since all the category pages are NoIndexed. So in theory, the only way Google possibly crawls that page (if it even does) is based on the fact that we submitted it to index and if that particular product editorial review URL gets any backlinks that Google crawls. Yet, we get omitted for that URL for certain KW variation in the SERPs. Although looking at Search Console, shows that URL as being indexed. Does all that change your stance on things or do you still believe that we're suppressing that particular URL by keeping the category pages closed off to Google?

Thanks!
2:47 pm on Oct 12, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


Have you thought about using schema for this content? Especially for reviews, schema can help each review to be found. It is simple to indicate the importance of the category pages: By not including them in your sitemap, and without any robots metatag at all, you are not including the category pages, but they may easily be indexed and they will be crawled. Let Google decide on the category pages and look into using Schema markup in the headers of your reviews. I would not noindex pages that contain the navigation to the reviews, even if that's all they have if that is how visitors find those pages.
4:16 pm on Oct 12, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy – yes, we currently do have the individual product page reviews marked up with schema. We mark up the overall aggregated average rating, review count, the product description overview, the product image, in addition, we mark up each individual user review (the author, rating, date, and actual review) since the top aggregated rating is comprised of the average from consumer reviews. Was that what you're referring to?
4:26 pm on Oct 12, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


Yes, that is the Schema markup I was referring to. Google offers a Rich Snippets schema markup testing tool [search.google.com] you could find useful.
5:12 pm on Oct 12, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14803
votes: 466


...since all the category pages are NoIndexed



That's a bad idea.
You could do a noindex,follow, but even that's a bad idea.

Category pages are GREAT pages to have indexed. They are an essential part of a site architecture that helps Google understand what a web page is about. The context of a top level category page helps define what the more granular individual pages are about.
6:50 am on Oct 13, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


@not2easy, thanks! Yep, just checked and all appears marked up properly for the product review URLs, with 0 errors and warnings.

@marinibuster – thanks. appreciate it. I guess it's safe to say that we began noticing a rapid rate of particular product review URLs get omitted from search results for certain (not always all) KW variations around the time that we noindexed, followed each top category page. But even back in Dec 2017 it was confirmed that eventually google stops indexing and crawling the URLs where even it has a noindex, follow. We were originally advised to add a noindex, follow on parent category pages to eliminate the ratio of potentially low quality pages since those category pages don't have much unique content other than just being used for organizational purposes.
3:09 pm on Oct 14, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14803
votes: 466


...since those category pages don't have much unique content other than just being used for organizational purposes.


Yeah, that's what I suspected and originally wrote then removed from my post.

That's the myth of the duplicate content "penalty" being merged with the "thin pages" penalty fear. WordPress SEO Plugins actually placed No Index, follow in category pages as a default setting. Which is where I first noticed this poor SEO practice.

The truth is that category pages are important. It's possible for them to rank for general high traffic pages, but that's something that is changing at the moment, related to how Google ranks pages.

The pages that the category page links to are the longer tail more specific pages for ranking. Google often prefers to rank those.

There's quite a bit of misinformation in the SEO industry. Even something as well accepted as "content cannibalization" or "keyword cannibalization" is total BS by virtue of the fact that it is such a general diagnosis. It's like your auto mechanic telling you your car is suffering from engine trouble. A more useful diagnosis is that your alternator needs replacing, right?

That's the big problem with a pop-SEO myth like Content Cannibalization. What is REALLY troubling those sites are things like thin pages (actual thin pages!), outdated content (which can be improved rather than pruned, improvement IS an option), and poor site architecture.

Those are THREE specific diagnoses of what can be wrong with a site. Once you know what's wrong, you can know your options better, like replacing your alternator. Engine Trouble/Content Cannibalization does not tell you anything and people should stop speaking of it because it's unhelpful.

Getting back to YOUR issue of the No Index,Follow, that too has it's roots in a general diagnosis being applied wholesale, including to situations where it should not. General Diagnoses are a poor way to SEO and result in these kinds of poor choices.

Good luck!
;)
Roger Montti
3:33 pm on Oct 14, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Apr 1, 2016
posts:2184
votes: 579


It is probably worth noting that Google has said that when using "nodinex follow" the links will not be followed given that the links will not be in the index, so "noindex follow" is really equivalent to "noindex nofollow". Also remember that indexing and ranking are two different things. Just because a page does not rank does not mean that there is no value in having it in the index.

Here is the link to the discussion on Google not following "noindex follow" links:
[webmasterworld.com...]
4:15 pm on Oct 14, 2018 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4017
votes: 246


It is probably worth noting that Google has said that when using "nodinex follow" the links will not be followed given that the links will not be in the index, so "noindex follow" is really equivalent to "noindex nofollow".
It's that "follow" is not a real directive for robots, neither is "dofollow". They may be used commonly, but completely ignored because "follow" is a default (for Googlebot at least).

It is somewhat OT here but worth noting that in the majority of cases, WP category pages should be noindexed. The site that HereWeGo123 is discussing is not a WP site, but that setting in WP is a feature most WP sites do want because they often assign more than one category to a given page. It is not a good practice to index the same content at multiple URLs on a site when the only difference is /category/ in the URL. The exception is when/if the /category/ is part of the permalink and pages are not assigned to multiple categories.

7:30 am on Oct 15, 2018 (gmt 0)

Moderator from US 

WebmasterWorld Administrator martinibuster is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 13, 2002
posts:14803
votes: 466


It is not a good practice to index the same content at multiple URLs on a site when the only difference is /category/ in the URL.


That's not the case in WordPress anymore.
It doesn't matter how many categories the page is associated with, the permalink URL remains the same and there's a canonical assigned to the page by default, no plugin necessary.

So a url example.com/blue/widgets/ that has a category of /reviews/ in addition to /blue/ will still be located at example.com/blue/widgets/ when you click through to it from the /reviews/ category.

Additionally, I believe the concerns about duplicate content are overstated with regards to top level category pages. As long as the category pages show excerpts, it's fine.
7:42 am on Oct 15, 2018 (gmt 0)

Junior Member

joined:Aug 9, 2017
posts:75
votes: 6


Thanks @martinibuster, @NickMNS and @not2easy – Extremely helpful and is forcing us to take a hard look at our current setup. Much appreciated, we have our takeaways! :)

I had another question that I wanted to ask, but will first explain our current structure to formulate the question, as that may also contribute to the issue of certain KW queries get omitted from the results. Our current product review page is set up as follows (we were advised):

A single product page has an article authored by us, and right below are consumer reviews. Once 2 or more consumer reviews enter for that particular product, they are on the same page below the article and it triggers sorting parameters for users to be able to sort customer reviews by highest rating, most recent, most helpful and lowest rating. The sorting parameters add a parameter in the URL, but the canonical will be https://www.example.com/produt-reviews.

Once 12 or more customer reviews are present for that product, pagination is triggered, and that obviously adds a new parameter should the user navigate to page 2 and beyond (there are 11 customer reviews per page, witht he first 11 being placed beneath the article we authored, but on page 2 and beyond, it's only 11 customer review per page.)

If NO sorting parameter is present, the canonical on page 2 and beyond will be https://www.example.com/product-reviews?page=2 and the rel tags for prev=https://www.example.com/product-reviews and next = https://www.example.com/product-reviews?page=3.

If a sorting parameter IS present in the URL while the user is on page 2, the canonical will still be the URL with pagination without sorting parameter, but the prev/next tags will contain the pagination parameter + sorting parameter (e.g. the user is located on page 2 with a sorting parameter enabled https://www.example.com/product-reviews?page=2&ord=highestrated).

While the user is navigating to page 2 and beyond, the original product article authored by us is no longer present (it's only present on the main root page of the product review page, to avoid duplicate content issues).

So to summarize, if sorting parameter IS enabled, the sorting parameter will be part of the URL in the prev/next rel tags but the canonical will never have it and will only contain the page number reflecting the current page the user is on within our pagination sequence.

I was unable to find a general consensus. Some documentation says to do that while others say to remove prev/next on pagination pages containing the sorting parameter and to in fact NoIndex the very URLs that contain the sorting parameters. Because essentially, when adding those sorting parameters, it does not change the content, only rearranges the order of the consumer reviews.

Do we have it right or what would be the best practice given our scenario?

I know it's a lot of info, but just a general rule of thumb?

Thanks!
This 35 message thread spans 2 pages: 35
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members