Google is stubborn it ranks keyword to wrong page

[Globetrotter]

I have a problem with Google ranking a high competitive keyword for the wrong url in the SERPS. Which results in high(er) bouncerates and low(er) time on site and a user experience which could be so much better, for this keyword.

I would like to rank for widgets to my widgets overview url (example.com/widget) which shows all the things you would need to know about widgets and related topics. For people visiting my website by typing in the main url would click the widget keyword in the main menu, and start reading from there.

Unfortunately Google does not want to rank the overview url. But instead is ranking a specific article within this widget section of my website which has a completely different topic but happens to be a related to widgets. (example.com/widget/article.html)

I get many views on this keyword according to WMT, but it is showing the “wrong” page, with a title which is about the subject of the article (about a different subject) instead of the keyword, so it doesn’t get too many clicks. The title of the article is related but not close enough to make sense for this keyword.

Then my visibility drops and a couple days later Google tries it again. This process is repeating for months now.

I’ve removed all unnecessary information / links / etc for the whole section thinking some of this information may possible muddle the indexation process. I tried to noindex the article, Google then shows an even more ridicules result and just picks another article.

This page which is ranking doesn’t have (much) links and the country overview url has got thousands. Internal and (natural) external. I tried to add a couple more to see if this helped but it didn’t.

Removed a lot of internal links but this caused the widget keyword to drop hard (still recovering).

How can I make Google rank the right page for the right keyword?

Is it allowed to redirect all visitors who come from Google (or any other search engine) on this article with this specific keyword to the right page?

[tedster]

Once this "wrong page" process starts, my experience has been that it can be a long struggle. My best success in this area came right after I added a lot of semantically related text to the "right page". Not just synonyms and stemmed versions, but a few strongly co-occurring terms and "n-grams" of various lengths.

[Globetrotter]

AT this moment i do not have any control over the content on the overview page. This is because it shows the latest updated or newest article on top.

You recommend to add some static content and optimize it for the keyword? Should i only show this static content or should i mix it? How long do you recommend this text should be?

[mark_roach]

I have had this problem with a number of pages on my site for years, so I agree with tedster than it can be hard to shift.

People search for widgets and google shows the widget makers page. eg. a search for "eggs" would return a page about "chickens"

I thought the most likely cause of the problem was because I had over-optimised the widget page. For example Title, Headings and Internal Links all using "widgets" as well as widget being mentioned on the page more times than is probably healthy.

Unfortunately like you I do not have a lot of control over the pages in question as most of my content is user generated.

I have had a little bit of success in de-optimising the elements of the page I have control over but for many of my categories the widget maker page is still king. Annoyingly a search for "widget makers" usually returns both the widget maker page and the widget page, unfortunately the search volume for that term is only 10% of the volume compared to a search for widgets.

[Globetrotter]

Over optimization might indeed be a cause of the problem. I thought about this aswel. I use the same backlinktext on the whole website to link back to the category page also with a lot of user generated content.

I tried to reduce the number of internal links but that was not a succes, maybe I should find a way to diversify the backlink text.

[hairycoo]

This is very interesting as I have the same problem on several sites.

Not just synonyms and stemmed versions, but a few strongly co-occurring terms and "n-grams" of various lengths.

Tedster, can I ask what you mean by the above and how do you figure out which are these co-occuring terms? Is it based on competitor research?

[realmaverick]

I have had this problem in the past. And in fact just made a pot about it.

In the past, i knew how to fix it. It was neary always related to, too many links ON the page. Strange I know. But do a count. For me anything over 80 was preventing some pages ranking.

One example, I had a page for blue widgets, no matter what it wouldn't rank. And instead a lousy page. Went on for over a year. One evening I removed half the links on the page and less than 48 hours later, it had a number 1 ranking.

But recently, a new page im having issues with, doesn't appear to be related to links, as I removed a large chunk of them but no move at present.

This is extremely frustrating.

[hairycoo]

I wonder if it could be the history of the "wrong" page. The "right" page seems to often be a newer page created to be more relevant for a particular search. But the "wrong" page seems to be so well established in SERPS that it's too much of a competitor for the new page to outrank.

Another theory of mine (again based on gut instinct) is that something about the "right" page offends Google in some way. But Google knows that the website is the most relevant result for that query so it goes for the second most relevant result. In my case I can't say it was the number of links.

Thoughts?

[deadsea]

At one point this was such a problem on my site that I implemented:

"I see you searched for widgets, but this page is about bidgets. Click here to see the widgets page."

Users almost always clicked over. I think that Google corrected itself by watching the behavior of toolbar users on my site.

[Globetrotter]

I can rule out the age of the history thing. My category url / page is way older then the article.

I will now try to add more content about the subject and reduce the number of links on this page.

If that doesn't help i will try deadsea's option.

[jaffstar]

Work on your inbound and internal linking strategy and you will solve this instantly... Been dow this road many times and you may have the right TOPIC but incorrect reputation based on linking.

[tedster]

Tedster, can I ask what you mean by the above and how do you figure out which are these co-occuring terms? Is it based on competitor research?

"Co-occuring terms" is a long established concept in IR (information retrieval, the art/science that evolved into web search.) The idea is that there will be some words and phrases that commonly occur together in documents about any topic - think "doctor's office" and "blood test", for example.

Those two example phrases are "2-grams" - a phrase of "n" words long is called an "n-gram". Google's patents about phrase based indexing [webmasterworld.com] lean on these concepts intensively. They process their entire corpus of web page data regularly to identify the phrases that are in actual use, and to identify newly emerging language conventions.

The absence of co-occurring phrases in a document can be a sign of "over-optimization" - too much focus on just the targeted keyword phrase in the content. Likewise, the presence of too many can be a spam signal. Patching together a lot of scraped sentences from many websites often creates this situation.

-----

Years ago, there used to be an online service (it's now gone) that took the top 50 results for any phrase and calculated the 2-grams, 3-grams and 4-grams that occurred in the top ranking documents. After using it a bit, I found that correlated phrases could be chosen rather intuitively by just poking around - reading the top pages, using various keyword research tools, the related search functions and so on.

For my purposes, I saw that I don't need the heavy duty calculations - just a handy jolt that pops my content writing out of the old-skool, lock step focus on "keyword phrase." In other words - more like the way normal, non-search-geek people would write about a topic.

[Robert Charlton]

AT this moment i do not have any control over the content on the overview page. This is because it shows the latest updated or newest article on top.

Under these circumstances, it's not reasonable to call Google "stubborn". Probably better to say that Google isn't psychic. ;)

I myself would definitely add some static content to precede links to the articles, and if possible, to avoid duplication, I'd use article summaries... ie, brief article descriptions... rather than quoted article excerpts.

jaffstar is right on about the importance of internal and external linking, but I wouldn't rely on linking alone. The overview page needs to be about "widgets" to a degree that its content, inbound links, and the articles that it links to all combine to make this page the nexus of "widgets" on your site. The page itself needs to be worth linking to.

With user-generated content, without some editorial control, it can be difficult to have enough consistent content for Google or your users to understand what the page is about. Most of the blogs, eg, that I see doing well are well edited.

[realmaverick]

Ted, that's very interesting. The top 10 "Galleries' that have now taken my rank, have NO TEXT. Absolutely none. I added text to all of my pages, but reading it back, it's definitely over optimised and I think that's caused the issue.

Are there any tools or further guides you recommend Ted? I'm going to experiment with the problem page and report back.

edit: actually, looking at my pages, even top ranking pages, there are lots of potential issues.

For example, on a gallery page, the links have there might be 50 links to widgets. Green widgets, blue widgets, orange widgets. Each has the word "widgets" in the anchor text. I've now changed it to "blue", "orange". Which actually makes more sense.

The opening paragraph of the page, also started with the target keyword. All of that combined with the url, title and h1 being a perfect match, was a little OTT.

In my experience, when the issues are fixed, the page quickly re-ranks. I'll update this if the page comes back to life.

[realmaverick]

Work on your inbound and internal linking strategy and you will solve this instantly... Been dow this road many times and you may have the right TOPIC but incorrect reputation based on linking.

Are you talking anchor text or suggesting that the target page, isn't as well linked?

[tedster]

Are there any tools or further guides you recommend Ted?

Just that you should go lightly - this can be like working with high voltage line. Different markets seem to have different criteria at work, different measures of what is statistically significant, so it's good to experiment gently.

In my biggest problem case, I added three phrases. The first two seemed to have no effect, and then I added a new phrase in on-page anchor text - and things resolved. Looking back, I had been an idiot not to use that particular phrase in the fisrt place.

In another case, one added phrase did the trick. And for some pages that were severely over the top, I also backed off the target phrase repetitions as well as adding in the new phrases.

[edited by: tedster at 9:21 pm (utc) on Sep 28, 2011]

[Globetrotter]

Work on your inbound and internal linking strategy and you will solve this instantly... Been dow this road many times and you may have the right TOPIC but incorrect reputation based on linking.

What does this mean if you would have to translate this into actions / changes?

Under these circumstances, it's not reasonable to call Google "stubborn". Probably better to say that Google isn't psychic. ;)

My bad :) I assumed that if google is able to predict what I’m looking for they could also detect where a page is about :P . But Seriously: I’ve done some more research and Bing is able to rank the correct page with the correct keyword. I'm in the top 10 for this keyword on Bing.

I did find in one of my notes that one of the online tools (ends with a Z :) ) i used and crawled that page said it might be over optimized. This would be because of the titels and the summery which are shown on this category page and are highly optimized for there own page and it might be a little too much.

So on the category page you will see a title (is also a link) and a short description about subjects related to widgets. They are user generated but validated before put online. The thing I do not control is which summary of an article shows up in what order on the page.

I’ve now added 290 words about subjects which are related to this keyword and reduced the number of summaries.

Let’s see what this does.

[realmaverick]

Globetrotter, I also used the tool from the site that ends with Z. Assuming it's the same one, does it give you a grade? All of mine are currently A. But I'm not sure whether this particular tool, is the one you're talking about and whether it would warn me of "over optimisation".

[realmaverick]

Tedster, just noticed your response. Thanks a lot.

[realmaverick]

I must say, I'm struggling with this whole "co occurring terms". I've just finished reading a huge thread about LSI and Keyword Co Occurrence (not the same thing I know). I get the concept. But what I'm struggling with, is how to be sure, the co occurring terms I'm choosing, are what Google would expect to see.

It appears more obvious when I make up examples, but when I take my particular niche, I'm struggling. I wish there were tools available. I notice Google wonder wheel is gone. Quintura appears to have stopped working.

What a pain in the backside.

[edited by: realmaverick at 9:45 pm (utc) on Sep 28, 2011]

[tedster]

But what I'm struggling with, is how to be sure, the co occurring terms I'm choosing, are what Google would expect to see.

That's just first date jitters. Or more seriously, if the vocabulary you're adding is clearly related to your page's topic, it's all good.

Yes, the wonder wheel used to be a helpful aid - it really stimulated some ideas for vocabulary expansion. But alas, it is now gone. I heard rumors that it had more scraping hits than real search users.

[realmaverick]

That's just first date jitters.

That's true. I never feel comfortable doing anything, until I 110% understand it. I guess a little more research and practice, will go a long way.

You're a real inspiration Ted :)

[Globetrotter]

I'm still having the same problem, my changed did have no effect. (Adding content en removing noise).

It might be that my menu is causiong some noise but i'm not sure.

@Tedster en Robert, i've send you the url and the keyword so you can have a look if you have the time.

What could I try next?