SEO Methodologies : A brief history of our industry:
1995) The early days of Yahoo.
Optimization was born out of the roots of AAA, A#1, and Acme style yellow pages/white pages alphabetical optimizations.
1996) Blind luck and keyword seasoning to taste.
The early days were stabs in the dark using simple keyword seasoning. Poke it here, and look for a reaction there. The first concepts of density and location started to be used.
You could still get a site listed in Yahoo by merely submitting it. As long as it wasn't too gaudy, you were in within 72 hours.
Late 96) The first papers begin to appear on the web about text matching, data mining, and interviews with se programmers.
Light bulbs of understanding begin going off around the early seo community. People began to realize just how databases work to match text and how they would be applied to the greater database of the web.
1997) The first algo crackers appear.
If it's a machine, we don't need to test it by blind experimentation, we can decode the algo mechanically. The first algo crackers were quite rudimentary by simply studying the make up of pages in the results many of the major clues to the algo's could be understood.
More specifically, several seo's decoded all 35 parameters to Excite and were able to build pages precisely to the algo; thus, generating #1 pages at will.
The first major "page jacking" and "bait and switch" incidents begin to happen. SEO's get code stolen and copied.
Mid 97) Several se's begin using Yahoo as a QA check. Thus, getting into Yahoo became paramount. Yahoo is flooded with submissions. Best guess is they processed less than 5% of submissions in 97 and 98. Impromptu Yahoo flame clubs formed anywhere there was a discussion about promotion.
Se's begin waking up to the fact that their sites are "portals" (in one door and out the other). Se's begin their first attempts at keeping people on the site in various ways. Some were intentional algo manipulations designed to keep people around the se and searching longer than they should have. (There are some big time stories here if any se techs would like to talk)
Late 97) Along came Infoseek's daily refresh. Submit it by 8am and you were in the db and pulling referrals by late afternoon. It was the first time "joe optimizer" could play the game without being a programmer. SEO explodes as people began to see simple and easy results in 24hours on Infoseek.
Spam becomes a very serious problem for the SE's as unscrupulous spam sites began to understand algos and how to manipulate them. Hotbot and Altavista were next to useless in late 97 due to spam (last half of 97 and most of 98 were the dark ages for se's).
The first "clustering" of results appears and has a major affect on algo decoding.
More page jacking incidents happen regularly. Hardly any top SEO doesn't have top ranked pages stolen and copied. Often copied into foreign domains out of jurisdiction.
Algo crackers begin to talk about the first cloaked pages appearing in the insurance and auto sectors. I am captivated by it.
Referrals begin to skyrocket for SEO's. 1k, 2k, and even 5k per site per day is not uncommon.
98) Let's get serious.
After several papers were delivered at the WWW conferences, it became clear se's were going to move to Off-The-Page criteria. Prerequisites such as link pop, directory listings, and listings age were going to be main parts of the new algo's.
Decoding algo's became very sophisticated in mid 98 and 99. Several optimization firms hired programmers to write efficient algo crackers.
It is also the first time I know of where a search engine used multiple algos for different top ten positions. Just because you could figure out what make a page #2 doesn't mean you have a clue about #3 which was positioned using different criteria.
The big push of "shop the competition" is born as several se's use the old "tell on your neighbor" ploy to clean up their results because their algos couldn't.
Page jacking and site theft is rampant. You can't put a top page on Altavista without it being stolen. Entire sites are mirrored as a means of "bumping off" the competition due to alta's horrible dupe page detector. Much the same occurred with Inktomi.
The big rounds of submission spamming wars begin as people spam the submit urls with your pages. Some say it worked for several years to get competition banned in the se's. Finally in late 98 se's begin to understand what is happening and put a stop to it by limiting submissions.
Se's begin to modernize with multi-languages, word lists (term vectors), and other language expertise - the era of the word guru is born.
Google hits the scene in earnest. Their first build of 25million urls makes it clear they have a future. I review it and am the first (beep beep) to propose link programs. People begin thinking in earnest about link pop and how to effect it.
Spam page/doorway page auto generators show up on the web every where and some are very good.
Referrals hold steady for those that know the game and stay off the radar. Using quality seo - that doesn't look like seo - rules the day.
Hello ODP! The first independent, free, "open source" directory is born. They represent a huge threat to the traditional directories. Out of "no where" comes the first ODP flames at a time when everyone was in love with the ODP (was it an anti-odp plant by a competitor, or was it real? You make the call).
Late 98-early 99) Altavista fights back with "too many urls" and bans huge segments of sites and sites with auto doorway page generators. Other engines begin out-and-out wars against seo. If a site said "we optimize" or "we promote" anywhere on it, they were banned in massive quantities. Much of that same mentality still exists today in many search engine offices.
Many seo firms begin falling out of the search engines in record numbers. Hardly any seo firm isn't affected. Loss of rankings on entire client lists is common. This is why you find old pro's who never talk about clients or link their websites with clients and why those that now know algo's cold - rarely talk in those terms.
Although the algo crackers are at their peak of performance, their utility falls as off-the-page factors such as link popularity become main stream in the se's. Decoding what makes a page top ten has never been more difficult. Those that know, now spend 10 times (literally) as much time to acheive half the rankings they did in 98. Algo crackers are not much more than statistic generators now.
Google's PageRank begins to bear fruit while the other se's self destruct under management chaos and mountains of red ink.
The Hubs and Authorities model is clearly a winner at Google. It universally clears out junk from the bloated db's and identifies the core mega sites in each keyword sector.
Although there has never been more competition, referrals hold steady through 98 and into early 99 across most of the engines.
Cloaking becomes almost mandatory on many se's to protect rankings and code. It is unfortunately used by those not so interested in those factors and more interested in spamming for the sake of instant successes.
Late 99) The effects of the end of SEO begin to sink in.
Goto begins to make it's major push. SEO's begin ppc'izing their billing with store-front redirect sites showing up every where.
Link pop schemes explode.
Other se's cut huge swaths out of their db's for unknown reasons. Part of it was size, some of it was spam, and some what just because they could.
SEO and traditional algo decoding techniques as we knew it, are all but relegated to the ash heap of history.
Referrals begin to plummet as competition sky rockets and the web matures. I secretly think 99 was when people "settled in" to a daily routine and began using search engines less and less. It was no longer this huge mystery that needed to be explored - they now used it to do productive things. eg: sites such as news take off in record numbers.
2000) A fairly deep shudder goes through the remaining industry as the end of what was left of Infoseek is gone.
The paid for play schemes and ppc schemes crank up in rapid succession in 2000. From Ink, to Alta, to even buying banners based on keywords - ppc and pfp is every where.
Meanwhile back in the real search industry, surfers look for an engine that actually works at finding them info - Google solidifies its position as the new defacto se.
The link pop craze of 99 begins to fade as it becomes very clear they are risky items - too easily tracked.
The last gasp for link pop programs is the building of fake awords programs, fake guestbooks, fake directories, and fake forum systems just to build fake link pop.
2001-) Bought and Paid For listings are everywhere. Goto is on all the major hubs from Yahoo, AOL, to even MSN. People abandon other se's such as Hotbot, Altavista, and Excite in record numbers. It's an exodus.
SEO is we knew it, is all but over. We are down to talking about the few remaining free specific engines and their systems. There is now a major difference in how se's work and how to "work them".
Welcome to the era of "All Google All The Time".
Many seo's have sleepless nights as we realize it is "Google or Bust".