Have you changed directories for index problems?

Google hates indexing or devalues pages in a particular directory...

[OldFaces]

Google can learn over time that particular directories aren't worth crawling or de-value the pages contained in those directories. I've witnessed some interesting things:

- pages contained in /dir1/ will have an index rate of 5%, but other directories (/dir2/, /dir3/, etc.) will have a more standard 75%+ index rate. *There are no technical on-page issues with /dir1/ pages that make them unique to other directories.

- when pages from /dir1/ are indexed, they typically will either be de-indexed, or outranked by the same domain's different directory (e.g. thin /dir2/ pages will outrank the superior content on /dir1/)

The way to solve this would be to 1.) fix the content contained in /dir1/ (remove garbage, de-dupe, and enhance remaining pages), then 2.) allow Google to see that /dir1/ is clearly the most valuable content on the domain.

The problem is; how does Google learn that /directory1/ has been cleaned up? It's already been trained to devalue /directory1. Give it time? How much is enough time? We've seen it's been nearly a year and Google hasn't learned.

We've tried manually submitting with Search Console the 10 URLs per day to show Google that our quality has improved. While some of the URLs will get indexed, if I check back a month later they are de-indexed. *Again, this is UNIQUE high quality (to any reader) content that Google currently has very little to no other information indexed from other domains. In other words, these aren't highly contested keywords and Google's index doesn't have much if anything about it from any websites.

It seems clear to me that /directory1/ has a big red "X" on it in Google's eyes.

My Question: Have you ever, or do you know of a situation, where you've switched directories to store your pages? If so, what was the impact on indexation after at minimum 2 months? Want to make sure when we discuss results that we remove the temporary 'honeymoon' indexing that is typical of G.

Yes, obviously 301 redirect, update sitemaps, internal links, yadda yadda. :c)

[OldFaces]

Oh and ps...yes I do see how it's kinda funny bringing this up just a handful of days after John Mu says "it's never a good idea to change URLs for SEO purposes."

The reality is that like with google support forums, the advice that is given directly from google and it's ecosystem is advice that's right for the majority, but that is not ALWAYS right for every situation.

[westcoast]

Yes, what you have observed is real.

I have done a number of experiments on this exact topic.

Google's either got a bug in their code, or they just assume "a directory that once was poor quality always will be" -- a poor assumption.

Our /dir1/ got penalized/demoted/low-pri'd 2 years ago for being thin content. Low crawl rate, low index rate, low SERP results... what you describe. We knew it was thin (links to other parts of the site), but it served a good purpose in the context of users on our site choosing different formats of content. Google was dropping the pages left and right though and clearly hated that directory. So 1.5 years ago we improved *every single URL*, moving all our main content into /dir1/ and removing 100% of those low quality pages. Should result in things getting better right?

Nope.

1.5 years later after that change and Google still refuses to even CRAWL 20% of the URLs in that directory, even though it has looked at the "new" stuff 1.5 years ago.

Moving that 20% of ignored URLs to a /dir2/ resulted in most of them being INSTANTLY crawled. Overnight. 95% crawl and index rate. Because the content is good.

Google clearly isn't updating the quality statistics for /directories/ and likely entire websites / domains in anything like a timely fashion, if at all.

Moral of the story -- until Google fixes their bugs / algorithms, once /directory/ gets a bad reputation it's unlikely to change no matter what you do. So move your stuff to /directory2/ to get some normalization to work around their bugs.

[edited by: westcoast at 5:04 am (utc) on Oct 7, 2022]

[westcoast]

"The reality is that like with google support forums, the advice that is given directly from google and it's ecosystem is advice that's right for the majority, but that is not ALWAYS right for every situation."

I think it's more the case of Google's advice is what would make sense if their algorithms worked as designed / expected.

In the end, Google is a set of highly complicated algorithms built atop massive amounts of data. It is easy to see that their software will have bugs (or unexpected/unintended results coming from machine learning), like any other software developer in existence, and that these bugs/design issues might not reflect what Google WANTS reality to be.

[OldFaces]

<3 to you as always Westcoast. Thank you. And yes, I agree with what you said as well with a 'highly complicated algorithms built atop massive amounts of data'.

Kudos to you for moving 20% of URLs and investing the time to ONLY do a subset. It's hard for us to only do a percentage because we have sitemaps, internal links, etc. to update. We like tidy code :) Anyhow, if I saw overnight what you did I'd jump right into moving the remaining.

I believe you mentioned you haven't moved over the remaining 80% because it is slowly getting indexed now? After 1.5years...

[tangor]

Internal links are a problem as regards user access, but sitemaps are not that serious. Many sites do not have sitemaps and experience zero problems.

NOTE: a sitemap will not correct the described behavior, but moving/changing urls WILL have an impact.

[Sgt_Kickaxe]

- moving/changing the urls hits right in the trust bucket and can hurt for months, avoid when possible.

- Besides title and content there is little that has a huge impact, you don't even need a page description with the major search engines. It's recommended you write one but it isn't critical, search writes one for you and it changes based on query(bonus!). Content > Title > Canonical > Some Schema markup types... the rest have diminishing returns. YMMV.

As for your specific issue, John Mueller covered the topic and suggested improving the category page itself. It needs to be indexable and offer some content to let G know what it's about if you want G to trust the links on it. His words.

It feels more complicated than that, you can noindex a category page and still get traffic to the pages it links to, but he has good basic advice of the might not help but won't hurt variety.

[OldFaces]

Tangor thanks for chiming in with "moving/changing urls WILL have an impact." Hearing people's experiences is hugely valuable and very much appreciated.

I agree, and DO know it'll have an impact...I'm just curious if it'll have a positive impact greater than all the obvious negative ones.

Sgt_Kickaxe:
- "avoid when possible". 100% Agree. It's something that I've never considered because of all the negatives. We are at a point now where, after many years, we've crossed off everything on the check-list that numerous smart minds have been able to identify as 'areas of improvement'.

- "Content > Title > Canonical > Some Schema markup types" is what matters. Exactly.

Our problem is not with the category page. In other words, /directory1/ contains all of the main subject matter. What's happening is that /directory0/ ('category' page kinda..we are educational not sales) is being indexed. /directory2/ which is supplemental information to individual pages in /directory1 is being indexed.

But /directory1/ pages which the site is all about is not being indexed. FWIW the actual url /directory1/ is indexed. But the many pages it contains are not.

[christianz]

Yes, Google does penalties at sitewide, directory-wide and page level granularity. That's according to my observations. Yes they are slow to update those quotas/penalties. But breaking links because of it doesn't seem like good tradeoff.

As for "thin content" being bad - it of course makes no sense to human beings. If thin was bad and thick was good, we would not have websites at all. Entire Internet would be one giant single page site, multiple petabytes large. And we would indulge in this orgy of thickness and everything would be great.

Sometimes the thinner the content the more it makes sense and the better it is for the users.

Also, the very purpose of Google is to "thin down" content and give us only what we are searching for.

Does Google demote "thin content" pages for no reason other than not having bunch of worthless filler text? Absolutely! But Google does many stupid things. Almost everything they do is stupid these days.

[tangor]

What's happening is that /directory0/ ('category' page kinda..we are educational not sales) is being indexed. /directory2/ which is supplemental information to individual pages in /directory1 is being indexed.

Are these pages being updated/refreshed with new info, etc?

Among the things g (and other SEs look for) is creation/modified indicators. If the pages are static not only in content, but also mod date the SEs will labor over the "new" content and ignore the old content.

[OldFaces]

Thank you Christianz! Slow to update...yeah I hear you. I've heard from others that after 1.5 years G still hasn't changed their mind. Can we deal with that? I guess...I mean we've been around for over 20 years. But since I don't even know IF they will, I'd rather not wait 1.5 years to just end up having to do it anyway.

"Google does many stupid things"...it's rather alarming the quality of SERPS (revising queries leads to the same websites), the unreliability of Ad Manager and Analytics, the charging for GSuite products which was supposed to always be free for legacy, etc. I'll leave this conversation for the many other posts about this subject. Just want to say yes, as a sizable shareholder (to me at least) I am concerned with what I'm seeing.

Hey Tangor - yes, these pages in directory1 are being updated - and significantly more routinely updated than any other pages on the site all of which are being indexed. Now we do have a LARGE quantity, so not every page. About 500-1k of these pages are updated everyday (it's a combination of UGC and internal folks updating).

When these pages are updated, the updated date on page reflects the new date. You mention other SEs... I will say this - Bing LOVES all the changes we have done [we've straight up removed 37% of the thin/garbage/low value pages - some of which users weren't too happy about, merged about 10% of the remaining pages (they were valid duplicates), and made sure internal cross reference linking between supplementary directory pages is strong. We had problems.]

Anyhow, since those changes Bing has DOUBLED our index and tripled our traffic to these pages. Google? Not one shift in the smallest bit.

Let me just say I appreciate very much everyone chiming in. I recognize everyone's names whose posted having been around these forums for many years - this is actually my second account - I forgot my account made before 2008 :P

All points/questions are spot on. And we've spent years trying to figure out this issue hence I'm quick to reply with our findings. The one other slim possibility that could be impacting /directory1/ pages is that we are constantly finding hijacked and spammy websites copying some of our content. WestCoast, who also suffers from a similar issues with /directory1/ pages not being indexed, also has the same problem of websites copying their data.

Yes, these copied websites don't outrank us, and heck, are even hard to find in SERPS - You'd have to do some very deep (exact match, + play around with Tools > Past 24 hours -> Past month) to even find them. But the *potential* problem is that since Google indexes these pages that they are potentially seen as duplicate content on our domain. Yeah, I know...a long shot right? But it IS possible. I don't want to open the whole can of worms about 'negative seo' and if it exists....but duplicate content definitely does.


Anyhow, to stay on point: It seems the general consensus is that Yes, directories can get penalized and it can take an unknown yet long amount of time for Google to change their minds. Yes, there are costs to switching directories.

[tangor]

About 500-1k of these pages are updated everyday (it's a combination of UGC and internal folks updating).

Having UGC might/might not have a determining factor. Are the supporting (other directory) pages equipped with UGC? If so, then this is not an issue. :)

[OldFaces]

Right - all pages in all directories are equipped with UGC. So in our case, not a factor. :)

[OldFaces]

Hey WestCoast if you see this I'd love to know if you ever moved over all your pages to the new directory, and whether or not the lift in index that you saw when you moved the subset of pages 'stuck' and wasn't just the honeymoon index period.

For us, we have continued to focus on product creating an engaging experience. Our users are super happy, but Google continues to not index a darn thing. So looks like potentially after we finish this GA4 transition, wrap up some UX stuff, we will indeed finally test moving all our content to a new directory.

It was funny, I had an interesting 'chat' with both Bard and openAI about this. I assume everyone here has thought to chat SEO with Bard? Anyhow, the gist is that both AI's said essentially 'yes, search engines can learn to devalue and not index/crawl pages in a particular directory' and 'yes, that's an interesting idea I hadn't seen before about changing directories'

[Nutterum]

I have seen this behavior before. Two main factors that may cause it is :

1) Interlinking structure of the website and thus providing strength to the pages of dir1 over dir2 and close match cannibalization. Most evident for the second one are the plethora of gambling affiliate websites that rely mainly on SEO and organic traffic to earn cash (as they will never be profitable using Gads) . Often however the overoptimisation of content on the main category pages cannibalizes tangent pages in other directories. Example dir1 is for casinos in general and has detailed pages for different outlets with 4k words each page. dir2 has slots in them and interlinks to the casinos that offer the various slots. Since dir1 has 40%+ of the slots text already present in the general casino pages in dir1 , Google simply deprecates most slot pages outside of some niche ones that have general monthly searches etc. Hope the example here helps.

2) Scraping. The more websites copy-paste your content or competitors have 70/80% of the content being the same (e.g. mandatory manufacture description and images, how-to manuals or similar) you can get the same effect, where Google will rank the obvious search node winners (the manufacturers themselves in our example) really big resellers (read Amazon or similar) then show some guides/how-tos or comparisons , and then all the sites that write content in a similar fashion if these are being indexed at all. Fun fact. Google will show thousands of results but these sometimes are not even in the index, if you really decide to visit the page. Say you go to result 500, google will simply redirect you to there.

[OldFaces]

Interesting that the Search of the Record [search-off-the-record.libsyn.com...] recently discussed how 'eventually we might learn that URLs under that pattern (directory) hold content that people will never actually search for, so why would we index it?'

'we can apply that on the whole URL pattern, so /xyz might actually be treated differently than /abc, because the pages under /xyz tend to be lower quality'

So yes, gbot does learn to treat directories differently.

[sriram]

Yes, I have changed directories for index problems in the past. This is a relatively common practice among SEO professionals, and it can be a very effective way to resolve indexing issues.

There are a few reasons why you might want to change directories for index problems. One reason is that your current directory may be penalized by Google. This can happen if your directory has a lot of low-quality content, or if it has been involved in spammy activities.

Another reason to change directories is that your current directory may be too large or complex for Google to index effectively. This is especially common for directories with a lot of nested subdirectories.

If you are considering changing directories for index problems, there are a few things you need to keep in mind. First, you need to choose a new directory that is high quality and relevant to your content. You should also make sure that the new directory is not penalized by Google.

Once you have chosen a new directory, you need to migrate your content to the new directory. Once you have migrated your content to the new directory, you need to submit the new directory to Google Search Console.

It is important to note that changing directories for index problems is not a guaranteed solution. It may take some time for Google to re-index your new directory, and there is no guarantee that your new directory will be indexed more effectively than your old directory. However, changing directories can be a very effective way to resolve indexing issues, and it is worth considering if you are having problems with your current directory.

[Alexterior]

So many troubles on my head with the indexes! Anyway, I've never heard about the indexation in such details, thank you

[OldFaces]

Besides the changes necessary onsite (sitemaps, internal links, etc.) I'd think a major concern is that the value of the external links those already indexed old/xyz pages receive will diminish in value. In other words and to use old terminology: external links will point to /old/xyz and you'd need to 301 redirect to /new/xyz, and that additional hop looses value to the link.

I'd assume (maybe foolishly) that already indexed old/xyz pages would likely remain indexed, but due to external link value as said above, the SERPs could fluctuate negatively.

Anyhow, great points sriram and thank you for sharing.

When we entice gbot to crawl us more (e.g. through redesign etc.,) it seems they'll spend a few days crawling a portion of these pages, index some, then return to a low crawl.

My concern would be that we change directories, gbot does crawl us and index a portion, but at a cost to SERPs.