Rankings hit on specific directory pages only

[mppj]

This is very strange... And has happened since May Day.

On June 17 ALL PAGES from a specific directory have stopped ranking in Google... Just the one set of pages residing in a particular directory only. All the rest are OK.

For example:

example.com/ is still ranking
example.com/widgetscategory/* are still ranking
example.com/widgets/* are gone.

Everything checks out; robots.txt etc... And we have not updated anything major onsite.

What is even stranger:

1. Back at the start of May Day, the other directory /widgetscategory/* pages were hit but at that time /widgets/* were fine. They came back 2 weeks later and have since stayed.

2. Just over the weekend I noticed the /widgets/ pages back in... Though this only lasted around 2 days.

This site is a quite an authority in a competitive vertical and has held rank for years.

Anyone else have a similar story?

[tedster]

Hey, mppj - welcome to the forums.

I've read various Google engineers saying that penalties are possible at the directory level, but I never ran into it "in the wild".

Does site:example.com/widgets/ still return URLs?

[incrediBILL]

I suspect you're linking out to bad sites and those pages are now being penalized.

Time for some deep link cleaning [webmasterworld.com] of your site IMO.

[mppj]

Cheers guys...

Does site:example.com/widgets/ still return URLs?

Yes - all "penalized" URLs are still returned using site: command.

I suspect you're linking out to bad sites and those pages are now being penalized.

We hardly link out at all - we have only made a small few link exchange partnerships with other highly authorative and complimentary websites. No #*$! sites at all, but they are across a range of pages each... This is quite common - and these "resource links" actually add value to our page... Think that would be an issue still?

[tedster]

You said all the URLs from that directory are "gone" - Do you mean they are not even showing on the very deep search results pages?

[mppj]

They show for quoted on page or title text (ie. only when we're the only site to show).

Otherwise they do not show at all.

One other thing: sometimes the /widgetcategory/ pages are being shown instead (albeit a little further down).

[mppj]

Sorry, my mistake, some are showing between 400-1000. Most of the time in the supplemental results.

[tedster]

Most of the time in the supplemental results.

Sorry to keep asking you questions like this, but I really want to understand exactly what you mean. How are you deciding that a URL is in the supplemental results?

[mppj]

No problem...

When you get to the point down in the SERPs where Google displays:

In order to show you the most relevant results, we have omitted some entries very similar to the 531 already displayed.
If you like, you can repeat the search with the omitted results included.

[tedster]

And then, if you click on that link, you do see your URL?

[mppj]

Yes, that's correct.

Btw, you said at the top you've heard of directory-level penalties... Do you remember where you heard that? Or is there any references you can point to?

[tedster]

Sorry - I read too much I guess and I'm having trouble finding a reference.

However, the fact that your URLs are now hidden behind an "omitted results" link seems like your content may now have been published elsewhere on the web and that copy stole your rankings. Have you checked on this?

[Robert Charlton]

Another possibility...

example.com/ is still ranking
example.com/widgetscategory/* are still ranking
example.com/widgets/* are gone.

How's the nav structure set up in your site... and, in particular, are your /widgets/ pages receiving their nav links from your /widgetscategory/ directory?

Is there any duplication of content between the pages in these two directories? ...I'm thinking in particular of duplication between a description or anchor text on the linking page and the text on the destination page for the link.

[mppj]

However, the fact that your URLs are now hidden behind an "omitted results" link seems like your content may now have been published elsewhere on the web and that copy stole your rankings. Have you checked on this?

Nope, I cannot find anything like this.

How's the nav structure set up in your site... and, in particular, are your /widgets/ pages receiving their nav links from your /widgetscategory/ directory?

Yes, widget pages are linked from widgetscategory pages.

Is there any duplication of content between the pages in these two directories?

Yes, there is content duplication across both levels - the category pages list widgets each with a small snippet from their description (which is fully displayed on the widget page). These snippets actually form most of the content on the category pages.

[Robert Charlton]

Yes, there is content duplication across both levels - the category pages list widgets each with a small snippet from their description (which is fully displayed on the widget page). These snippets actually form most of the content on the category pages.

And your category pages are still ranking and your widget pages have dropped out....

While it's not unheard of for a category page to rank in preference subpage, since this wasn't your situation before, we have to think about what Mayday had to do with it.

I'm thinking it might have to do with the decreased effectiveness of internal links that's been observed in Mayday. This is probably making the snippets in the category pages so much more prominent than the widget pages that the widget pages are being seen as dupe content.

There are a lot of sites apparently struggling with the same issue, many resorting to rel="nofollow" PageRank sculpting to solve the problem. Nofollow... or blocking pages in general... is a very bad way, I feel, to deal with the situation.

In this discussion, eg...

No Follow to Homepage - Why are sites doing this?
http://www.webmasterworld.com/google/4163381.htm
[webmasterworld.com...]

...I comment...

After seeing some of the above examples and others, I'm thinking this may be an effort to keep category pages from ranking above a specific story page.... Many of the nofollow efforts I see are trying to control which internal duplicate content ranks....

Also see this thread...

PR Sculpting Doesn't Work and Internal NoFollow Can Harm Your Site
http://www.webmasterworld.com/google/4161726.htm
[webmasterworld.com...]

A better way to fix the problem, I feel, would be either to rewrite the snippets on the category pages, to reduce or eliminate the duplication, and/or to get more deep links at the widget page level. This latter might be extremely hard to do. Adjusting nav structure might also be an option.

Since this wasn't happening before, I'm positing that there might be a kind of a link juice "threshold" level for the widget pages to be viable.

One of the things I wonder about in this situation is what deep links to category pages might do... whether they'd make your category pages completely overshadow your widget pages, or whether they'd help your widget pages rise enough above the supposed threshold level to rank in spite of the duplication.

An additional question I should add... do you have similar duplication in other directories which are not being affected?

[mppj]

Appreciate the great response, Robert...

After seeing some of the above examples and others, I'm thinking this may be an effort to keep category pages from ranking above a specific story page.... Many of the nofollow efforts I see are trying to control which internal duplicate content ranks....

Ahh interesting and conceivable.

I have a question though - what's the difference between this and news, forums and blog sites that all have this kind of structure (ie. category pages and snippets)?

An additional question I should add... do you have similar duplication in other directories which are not being affected?

No, however as noted, the category pages first were hit by this directory level "penalty"... then back in after 1.5 weeks... then a few weeks later the widget pages were hit.

It almost seems like the algo is having a hard time trying to pick the "right" set of pages.