Forum Moderators: open
Thank you,
Ryan Allis
On November 15, 2003, the SERPs (Search Engine Result Pages) in Google were dramatically altered. Although Google has been known to go through a reshuffling (appropriately named a Google Dance) every 2 months or so, this 'Dance' seems to be more like a drunken Mexican salsa that its usual conservative fox-trot.
Most likely, you will already know if your web site has been affected. You may have seen a significant drop-off in traffic around Nov. 15. Three of my sites have been hit. While one could understand dropping down a few positions, since November 15, the sites that previously held these rankings are nowhere to be found in the top 10,000 rankings. Such radical repositionings have left many mom-and-pop and small businesses devastated and out of luck for the holiday season. With Google controlling approximately 85% of Internet searches, many businesses are finding a need to lay off workers or rapidly cancel inventory orders. This situation deserves a closer look.
What the Early Research is Showing
From what early research shows, it seems that Google has put into place what has been quickly termed in the industry as an 'Over Optimization Penalty' (OOP) that takes into account the incoming link text and the on-site keyword frequency. If too many sites that link to your site use link text containing a word that is repeated more than a certain number of times on your home page, that page will be assessed the penalty and either demoted to oblivion or removed entirely from the rankings. In a sense Google is penalizing sites for being optimized for the search engines--without any forewarning of a change in policy.
Here is what else we know:
- The OOP is keyword specific, not site specific. Google has selected only certain keywords to apply the OOP for.
- Certain highly competitive keywords have lost many of the listings.
How to Know if Your Site Has Been Penalized
There are a few ways to know if your site has been penalized. The first, mentioned earlier, is if you noticed a significant drop in traffic around the 15th of November you've likely been hit. Here are ways to be sure:
1. Go to google.com. Type in any search term you recall being well-ranked for. See you site logs to see which terms you received search engine traffic from. If your site is nowhere to be found it's likely been penalized.
2. Type in the search term you suspect being penalized for, followed by "-dkjsahfdsaf" (or any other similar gibberish, without the quotes). This will remove the OOP and you should see what your results should be.
3. Or, simply go to www.**** to have this automated for you. Just type in the search term and see quickly what the search engine results would be if the OOP was not in effect. This site, put up less than a week ago, has quickly gained in popularity, becoming one of the 5000 most visited web sites on the Internet in a matter of days.
The Basics of SEO Redefined. Should One De-Optimize?
Search engine optimization consultants such as myself have known for years that the basics of SEO are:
- put your target keyword or keyphrase in your title, meta-tags, and alt-tags
- put your target keyword or keyphrase in an H1 tag near the top of your page
- repeat your keyword or keyphrase 5-10 times throughout the page
- create quality content on your site and update it regularly
- use a site map (linked to from every page) that links to all of your pages
- build lots of relevant links to your site
- ensure that your target keyword or keyphrase is in the link text of your incoming links
Now, however, the best practices for keyword frequency and link text will likely trigger the Google OOP. There is surely no denying that there are many low quality sites have used link farms and spammed blog comments in order to increase their PageRank (Google's measure of site quality) and link popularity. However, a differentiation must be made from these sites and quality sites with dozens or hundreds of pages of informational well-written content that have taken the time to properly build links.
So if you have been affected, what can you do? Should one de-optimize their site, or wait it out? Should one create one site for Google and one for the 'normal engines?' Is this a case of a filter been turned on too tight that Google will fix in a matter of days or something much more?
These are all serious questions that no one seems to have answers to. At this point we recommend making the following changes to your site if, and only if, your rankings seem to have been affected:
1. Contact a few of your link partners via email. Ask them to change the link text so that the keyword you have been penalized for is not in the link text or the keyphrase is in a different order than the order you are penalized for.
2. Open up the page that has been penalized (usually your home page) and reduce the number of times that you have the keyword on your site. Keep the number under 5 times for every 100 words you have on your page.
3. If you are targeting a keyphrase (a multiple-word keyword) reduce the number of times that your page has the target keyphrase in the exact order you are targeting. Mix up the order. For example, if you are targeting "Florida web designer" change this text on your site to "web site designer in florida" and "florida-based web site design services."
It is important to note that these 'de-optimization' steps should only be taken if you know that you have been affected by the Google OOP.
Why did Google do this? There are two possible answers. First, it is possible that Google has simply made an honest (yet very poor) attempt at removing many of the low-quality web sites in their results that had little quality content and received their positions from link farms and spamdexing. The evidence and the search engine results point to another potential answer.
A second theory, which has gained credence in the past days within the industry, is that in preparation for its Initial Public Offering (possibly this Spring), Google has developed a way to increase its revenue. How? By removing many of the sites that are optimized for the search engines on major commerical search terms, thereby increasing the use of its AdWords paid search results (cost-per-click) system. Is this the case? Maybe, maybe not.
Perhaps both of these reasons came into play. Perhaps Google execs thought they could
1) improve the quality of their rankings,
2) remove many of the 'spammy' low-quality sites
3) because of #2, increase AdWords revenues and
4) because of better results and more revenue have a better chance at a successful IPO.
Sadly, for Google, this plan had a detrimental flaw.
What Google Should Do
While there are positives that have come from this OOP filter, the filter needs to be adjusted. Here is what Google should do:
1. Post a communiqué on its web site explaining in as much detail as they are able what they have done and what they are doing to fix it;
2. Reduce the weight of OOP;
3. If the OOP is indeed a static penalty that can only be removed by a human, change it to a dynamic penalty that is analyzed and assessed with each major update; and
4. Establish an appeal process through which site owners which feel they are following all rules and have quality content can have a human (or enlightened spider) review their site and remove the OOP if appropriate.
When this recent update broke on November 15, webmasters clamored in the thousands to the industry forums such as webmasterworld.com. The mis-update was quickly titled "Florida Update 2003" and the initial common wisdom was that Google had made a serious mistake that would be fixed within 3-4 days and everyone should just stay put and wait for Google to 'fix itself.' While the rankings are still dancing, this fix has yet to come. High quality sites with lots of good content that have done everything right are being severely penalized.
If Google does not act quickly, it will soon lose market share and its reputation as the provider of the best search results. With Yahoo's recent acquisition of Inktomi, Alltheweb/FAST, and Altavista, it most likely will soon renege on its deal to serve Google results and may, in the process, create the future "best search engine on the 'net." Google, for now, has gone bananas in its recent meringue, and it may soon be spoiled rotten.
Make it singular.
Search Engine -oewirysldskfjldajhf
Returns google as number one.
They are suffering from their own algo problem.
Second point...
If g=oogle can't get to number 1 on their own engine..
how are we supposed to figure it out. (tongue in cheek
and forefinger to the wind).
Cheers
[edited by: Chalupee at 8:15 pm (utc) on Dec. 5, 2003]
Work at Google
We can't hire smart people
fast enough!
www.google.com/jobs/
Regarding that content thing, i think Brett's 26 steps [webmasterworld.com] post shows the importance, and i agree... So, what exactly is this thing "content"? It can be so many things really. I'd recommend finding 5-10 top ranking commercial sites (yes, they are out there) and study those in detail. What exactly do these sites do to stand out from the crowd?
I can't really be more specific here, and i don't intend to offend anyone. As lawyers and doctors usually say: This is not to be considered individual professional advice...
Also, do i read "theming" or am i wrong here? Google trying to figure out what the purpose of your site really is? If that is so, then "sale" "buy" "shop" etc. might become relevant keywords to target along with product names.
/claus
btw. i think that famous knob has just been given yet another twist in some direction, it happens quite frequently it seems, so check those serps...
[edited by: claus at 11:31 pm (utc) on Dec. 5, 2003]
The black helicopter theory of the day: the algorithm change is designed so Google buys more adwords.
Hay, the Author Anderson guys had to go somewhere didn't they?;)
the algorithm change is designed so Google buys more adwords.Ha. I just clicked on the link 100 times. Think of what i just cost them. Now I feel better.;)
I haven't been hurt badly, but pushed down a bit
by the tiered listings I mentioned in another thread.
Asking in another thread about g=oogle and link popularity
(which was naive 'cause I know it's important) but
actually I was wondering if it has more weight in this
current situation...
The only thing I can make of the tiered effect...
2 www.thedomain.com
2a---www.thedomain.com/next-important-keyword-tothetopic-with-very-good-linking.htm
Is as you see above the next important keyword to the main keyword(s).
For example...
2. Donkeys
2a.----Donkeys pack animals (very well linked and important to first or main keyword)
Different concept now:
Maybe there needs to be more than one algorithm g=oogle should use... one for say real estate, another one for say scientific information... etc.
From all the complaints, it appears the one algorithm isn't doing the segregation properly. (please escuzz my naivete')
(added this in edit)
Oh yeah... anybody fooled around with changing preferences as to number of listings to display per page...
If I use 50 per page... I get 4-6 more tiered listings than if I just use 10 per page... so on the 50 per page people get bumped even further down the list.
Chalupee
Word is that the -ex datacenter has the results...
Good luck...
It wouldn't be difficult for Google to identify sites that were once filtered out by the OOP filter and suddenly are not due to deoptimization. Google could use this data to flag your pages as SEO manipulated pages and use it to weed out the false positives.
Solution: do nothing
While you are doing nothing, I will be busy working to make my site better for what visitors I have left.
Logically it would seem next to impossible to monitor changes in sites which have been filtered out for certain keyphrases as I just can't imagine them having the resources to do so. Perhaps someone who has tech skills might be able to illuminate us on what is and isn't possible.
In any event I think it's always wise to sit tight (if you can) and see what happens after the next "update" whenever that may be.
Again, the reason Google may want to do this is to weed out the false positives -- over time, the non-SEO'd pages wouldn't have changed significantly.
I built good sound html sites 4 years ago (that I have done little to since then) before I ever cared about Google that still perform well. Now these would be considered SEOed? Give me a break.
Lets not be so arrogant to think that Google feels threatened by a handful of people who build websites. Please, if you have to think conspiracy, go with Adwords. At least that one is bought by the media.
Lets not be so arrogant to think that Google feels threatened by a handful of people who build websites.
Wired, January 2003
[wired.com...]
Key quote:
The way Brin sees it, the optimizers are co-opting Google's bond of trust with its users. He regards optimizers the way a mother grizzly might regard a hunter jabbing at her cub with a stick."When we change and improve our technology, things get shuffled around," Brin says, "and sometimes it has a disproportionate effect on optimization sites."
Solution: do nothing
And if your site normally is actualized every day? new pages, pages taken off etc, content changing.
what should you do, stop working? google may think your site is SEO manipulated
Whether you believe or just want to believe in OOP (over optimized pages), it will eventually lead to the final destination PUS (pay up sucker).
The party is over, turn out the lights.
Time to make new friends. Where is AVguy?
The thing that I really do not understand is, why would google penalise SEO'd sites? This seems barmy as all we actually do is write content that litte googlebot can understand. Without us doing this, the results would not be worth much at all. Lets all design in flash, not put title tags in our code, all links in javascript etc shall we?
All most have done is followed Googles guidelines - yes?
Google likes the optimization of quality sites that users will appreciate in their search results.
Good thought, that's what I'm hoping for. Something sure knocked out a lot of sites though, (not ours). There really does seem to be certain SEO techniques that worked in the past that have recently been blown out of the water. So it goes. If you're selling stuff that a million other people are selling, then it's dicey from the start.
On topic: It's a bit over the top to imagine that Google is monitoring changes in sites to detect the removal of SEO gaming and then putting the sites on a watch-list... this way lies madness.
I am not talking about a conspiracy or any evil intentions. Google isn't going to sit still and be predictable commercial target.
So, they choose the biggest shopping season of the year to shake things up and maybe send a message to people who don't pay Google a dime for traffic or profits.
The fact is that Google already has all the "money word" data it could dream of. All of the advertisers gave it to them... logged onto adwords, compiling and assigning monetary value to each phrase. Filters developed for those phrases (depending on popularity of advertising popularity?) could be applied in varying amounts filtering the majority of the sites who targets sets of keyphrases. It makes sense that Google would change the rankings by filtering results in highly commercial areas with a surplus of traffic and advertisers.
This obviously brings more advertising profits and they would still be 'crusading against' being overrun by tactics designed to bring free commercial profit.
I hear webmasters say they have hundreds of missing index pages, hundreds of websites vanish.... I couldn't imagine my staff being able to handle more than our company's ONE! website. We spend time obsessing over the details of our content and adding useful features and information.
I wonder how dedicated a webmaster could be to dozens or hundreds of websites, we seem to have our hands full with ONE!?
I don't think Google has got it right yet but things are probably going to be different. Google is a commercial target and they have to keep moving. An average of 85 of the top 100 sites are gone across most phrases in our market. We are not one of them. Most of the "missing" sites were not spam sites, they didn't seem to break any of the "imaginary rules" but their websites were clearly targeting commercial key phrases.
I think that we need to get back to basics, think like a Google engineer and think of the big idea. I’ve floundered about looking at the microcosm of my pages trying to figure out a single answer that would explain all of the different and conflicting reports of artefacts and coincidences. I too at first thought that <h> tags and title tags which included the targeted words coupled with some mystery ingredient might be causing the problems that I was seeing.
I read about the kind of spam filtering technology used to filter out bad email and put forward a rational explanation for why this could never work in the context of a Web search engine because of the skewing of results which would lead to an unacceptable number of false positives in the results that really matter ie the first 10 or 20. I wouldn’t rule this out but I think that Google has a better solution than probability filtering.
If you read this article talks about Google Bombing and how this type of activity is ruining Googles results you will start to see what the real problem is for Google. [slate.msn.com...] In effect unless Google did something pretty quick its unique selling proposition would be ruined and rather than floating the business they might have to try and stop it sinking.
This article starts by talking about PR0, an earlier attempt by Google to stop the spam problem, but goes on to talk about BadRank and explains how this might work.
Search for "Google's PageRank 0 Penalty" exactly as here including the "" Ignore the PR0 stuff and scroll down to read about BadRank.
If Google applied BadRank to a page based only on certain keywords then they would have created a Web spam filter which uses their patented PageRank technology applied in reverse.
If I were a Google engineer it would really appeal to me to use exactly the same technology, the same premis of the Web being a uniquely democratic place to fight against spam. It so simple it is brilliant. If you can build up more PageRank by linking to and having links from authority sites then using exactly the same premis in reverse you could build up an unreported BadRank by linking to and having links from “bad sites of pages” which might be used in a number of ways within the algorithm that produces the final SERPs for a given term.
So how do you build a list of terms to target and how do you identify what a bad site is. I think that Google’s engineers would look for an automatic solution to this. I suppose they might have used a hand built list based on spam reports but it doesn’t fit with their way of thinking and it means that Google would always be playing catch up when they should have enough data to stay ahead of the game.
Here’s how I would do it.
1. Identifying terms to target. Google just has to look at what searches are being done. Whilst it might be fun to Google Bomb some stupid term if no one is searching for that term then it does no harm so it can be ignored. If there is a certain level of searching for a certain term and/or the growth in the use of that term reaches a certain critical level it is automatically added to the list.
2. Identify bad/spam sites. If a site has links out including a particular search term but no links in using that term then it is probably bad. If it has many links out each using terms in the automatically generated target list, and no or a very low level of links in using those terms then it is probably very bad. If it also does not have any authority sites linking to it and it has a high density of the amalgum of words and phrases in the target list then it is probably a spam site.
3. Decide on a level of back linking to spam sites that will trigger a penalty “Google's Matt Cutts' remark on this issue is: "If someone accidentally does a link to a bad site, that may not hurt them, but if they do twenty, that's a problem."” (Quoted from the page on bad rank mentioned above) What if just a few reciprocated links to sites which Google’s automated system has now identified as very bad indeed trips the switch and you site gets a temporary zero weighting on all algo factors for that search.
4. What mechanism would you use to store this information. In the same way that PageRank is stored following the monthly recalculation a Badrank figure could be stored but not reported. The match is then very simple if your site has a particular level of Badrank and someone searches for any term that is in the list being targeted at that time then it is filtered. If the term is not in the filter list then your site will still appear in its natural position for that term. The system does not need to store which terms your site/page is bad for, it can assume that it is bad for all of the filtered terms.
Bad site + filtered term searched for = apply null algo
Bad site + non-filtered term = apply normal algo
Good site + any term = apply normal algo
This form of rolling term specific filtering is scalable to the web, it produces a low level of false positives and it defuses most Google bombs before they go off. It is so simple it is brilliant, If this is not what they are doing then they need to stop what they are doing and start doing this as soon as possible.
Good luck
Sid
I promise this is definately my last theory on this topic.
Yup. On the first cut, it looks simple, which is how G does almost all the things.
Identifying terms to target. Google just has to look at what searches are being done. Whilst it might be fun to Google Bomb some stupid term if no one is searching for that term then it does no harm so it can be ignored. If there is a certain level of searching for a certain term and/or the growth in the use of that term reaches a certain critical level it is automatically added to the list.
Another way to identify the target terms is, the keywords for which Adwords are bid on (aka commercial keywords). This is on the proposition that almost all the spam occurs where $ are bid on.
Your step 2 might be something more involved. I am not sure of the metrics, whatever they maybe. But, it's intention would be to trigger the sites that artificially inflate their relevancy to search terms.
Very nice post, Hissingsid. Thanks for sharing.
Scratch that one. I have filtered and unfiltered within the same (large) site.
