Welcome to WebmasterWorld Guest from 188.8.131.52
I'm thinking its time to move ahead and stop waiting for Google to somehow get right, to one day see all my dropped pages reappear.
So what to do? Options:
a) move ahead with new work and just try to forget the dropped pages/content
b)start over in some way... post the missing content to a new site and hope for the best
c) what else? Any ideas out there?
I emailed G, and they claimed it was due to our host. This held some water, cause I found that the other site was using the same name servers. So about a month ago, we switched hosts, and although the G bot goes over our site every day, the site is still incorrectly indexed.
We write and submit articles, and I believe we are no longer viewed as the origin of content. So now we've probably triggered the G dup-content filters.
I could go on for days, but having done SEM for a few years, I know better than to freak out. Y!, and M, are still bringing the site great traffic. The clients are happy, and they're business is doing great.
IMO, G is missing out on providing their users with the best results out there. There's not really any competitors, in our client's site's niche, but for the main keywords, G now ranks us anywhere from 300 to 500 in the SERPs.
I have faith that it will all subside. But then again maybe it won't. After all, if I had a search engine, I wouldn't want anyone knowing patterns in my algo.
Besides, isn't there a direct correlation between adwords revenue for G, and sites losing their rankings in G?
After all, as webmaster's site's decline, they tend to dump more money into adwords in order to continue recieving traffic.
BD uses 64 bit processors, and to my knowledge, the 64 bit technology still has alot of bugs. If anyone's gonna come through it'll be G.
So let's all relax, go have a beer, or a lemonade or whatever, and let G do their thing.
I understand being in it for the long haul but loosing 95% of my sites pages in Mar & April and the being flat in May with only a hand full of pages left, this advice comes across as a little weak. All during this 3-month stint of looking for possible site problems and following clean-up recommendations with only continued page losses to show for my efforts.
Especially, when after making a simple navigation change on my site 2 weeks ago that moved critical pages closer to the home page and seeing thousands of pages indexed and traffic triple in the following week. All this because people have the opportunity to locate one of 22,000 pages of content this week as opposed to 150 pages two weeks ago. Continuing to build content when I had thousands of pages of content already not being indexed didn't seem to be where my priorities should be. Now that the existing pages are indexed I can go back to creating content, while earning a little additional revenue.
Maybe this is a short term fix (maybe not) but Adsense revenues are back for now. For me a quarter of a year is long enough to wait before taking action that bears fruit.
At the beginning of BD, like many here, pages in my site began to rise and fall ... they would be in the top 10 one day, and down at 20 to 30 the next.
Then, all of a sudden, Google went completely haywire. 95% of the pages in my site were dropped out of the index.
The ranking for the search strings I monitor daily varied wildly, with spam sites (hidden text, etc) getting very good ranking.
Now, some pages are slowly being reincluded and my site is 50% indexed again. However, the cache of some of these pages are months old ... in some cases, 8 months old.
One hour I am at the top of the index, the next I'm back at 20 to 25.
Also, there are many pages that I have always been coded "robots" content="noindex,follow" that are currenty showing as supplemental pages. Why are they showing at all? They have no value. Most of them are form pages, privacy statements, etc.
It appears to me that Google is trying to rebuild their index. Quite frankly, I think they had an "oh sh*t" moment and they nearly lost their index. Now, they have taken a few steps back and are trying to rebuild it as fast as they can.
Whatever the excuse, it is obvious that G has taken a huge leap back in time.
And, whatever the excuse, I see spam sites and sites with little or no correlation to search strings ranking high in the index. Why? Who knows!
What a mess.
Like "ask.com" says in thier television commercials ... "Just because Google has brand recognition does not mean they are better."
Indeed. For me, I'm off to brand X.
PS If you type in "blog" into google up comes over 2.3 billion pages. I thought they had 8 billion indexed a year ago so that is 25% of the index now.
It turns out that Google has started indexing our version of the Open Directory, in spite of a robots.txt file that kindly asks it not to and a <META NAME="ROBOTS" CONTENT="NOINDEX"> tag on every single directory page. Google has gotten some kind of search engine flu, that's for sure.
These points just prove that even if you are doing well in google, you are at their mercy. The search results they provide are unstable for the webmaster.
It is all very well saying lets pick ourselves up and learn from our mistakes and move forward, but you can't do that with any certainty because you are completely at the whim of google.
The way to move forward here is to diversify. Change your business model so that you can get traffic from other places. Make it so that any traffic that you get from the search engines is just a bonus. It may be gone again tomorrow. Don't rely on it.
Google will always be like they are now. They are just over engineered and that means the smallest change they make here or there can have devasting effects on where your site ranks.
Think of it this way - everybody has to take their turn hitting the skids in the serps :-)
These points just prove that even if you are doing well in google, you are at their mercy.
Sure, and if you own a hotel or restaurant, you may depend on traffic from Michelin, Mobil, or the automobile club guides. And if you write a book, it probably won't make the bestseller list unless it's featured in the major review media. So what? That's just how the world works. (If you can't get a Michelin star and THE NEW YORK TIMES BOOK REVIEW won't feature your novel, launch a PR campaign or advertise.)
Google will always be like they are now. They are just over engineered and that means the smallest change they make here or there can have devasting effects on where your site ranks.
Or maybe not. I suspect that most "over-engineering" is on the Webmaster/SEO side, not at Google's end. If your site is organic and has enough editorial diversity, a tiny Google change here and there may affect individual pages, but it shouldn't affect your site's referrals and traffic overall. (That isn't to say that #*$! can't happen now and then, with Google as with life in general.)
And if you write a book, it probably won't make the bestseller list unless it's featured in the major review media. So what? That's just how the world works. (If you can't get a Michelin star and THE NEW YORK TIMES BOOK REVIEW won't feature your novel, launch a PR campaign or advertise.)
You're right about the pr, but I think you are really wrong about the analogy. I'm guessing you aren't that familiar with the book publishing industry.
The flaws in the analogy are so many that I can't even begin to cover them all. But the prime one is the degree of monopoly. By and large it's book publishers who make or break books, because it's book publishers who GET you that review in the NYTIMES. It's publishers that get your book in the store shelves. However,there are a lot more book publishers than there are search engines like google.
If I don't like my book publisher's support, I go elsewhere with my next project. I can't go elsewhere with my next website. No google, no significant traffic.
BTW, I've never been on the NYTIMES best seller list, but one of my books has been sitting in the top several thousand sellers at amazon for over two years. I have a website that's about two years old that has had its traffic completely trashed since March, except for two days (go figure).
Google doesn't owe me anything (at least my publishers do, either contractually or otherwise), but the stark, harsh reality is that the WAY it is operating right now is making many online enterprises (for profit or not) completely pointless. That's not their intent. But that's what's happening.
I'm guessing you aren't that familiar with the book publishing industry.
I've worked in the publishing industry as a literary agent, editor, and author.
That doesn't mean you have to agree with my analogy, of course, just as I don't have to agree with your questionable suggestion that Google is a "monopoly."
I can't blame google for the fact that both msn and yahoo delivered such anemic search for so long, and that the new msn is so weak that even the msn guys now admit that it will be at least 5 years before they have a product that can actually compete with google.
So it's an odd situation in the search world right now. Yahoo could improve, they could get share, but they just dont' seem to care enough even though pretend to. And msn is another weak ms web venture, so don't hold your breath.
I would say, to keep the analogy fairly accurate, that it is in fact just about as hard to get published by a big house currently as it is to get listed on google - well, no, that's ridiculous, who am I kidding, it's WAY harder to get published than it is to get listed on google, by a factor of at least 10, probably more.
You can do it, but it's not easy. The main difference is that anyone who takes the time and energy required can probably get listed where they want eventually if they follow what EFV says and ignore all the typical WebmasterWorld google forums talk as much as possible. Or follow brett's 26 steps, whatever. Problem is that's hard to do.
Takes a lot of work, it's a many year process, and requires content that is worth other people's attention so they link to it. That's not your opinion of the content, it's other people's opinion of it. Everyone thinks their site is great and their content unique, as a writer you should recognize that problem, LOL...
I'm seeing exactly what EFV is seeing, and I'm not paying any attention to my page indexed count, my backlink count, whatever. The only change I saw in big daddy was that on two sites I finally started ranking for the target terms, one number 1 which eluded me for 4 years, another top 30, which also eluded me for many years. Given the heavily competitive nature of the top 30 term, and the fact that I do basically zero work to get links on that site, I am totally satisfied with that placing, it's way better than the > 200 it used to be.
The advice efv gives is good, as long as the site hasn't used spammy methods ever in its past, including fake link building etc. If it has, don't blame google for that.
By far the most accurate way to think of google is not as a search engine, but as an advertising portal that you would like to maybe get free placement on. It's very easy to get placed on it, just pay adwords.
This is hard to get for some people, but google is a media company that sells advertising, that's its business. They offer search not for you the site owner, but for their clients, the searchers, who have almost nothing to do with, or in common with, people like us.
To go with the writing analogy a bit more, I could sit around complaining that my writing hasn't gotten published, or I could find a way to get it published. One thing's for sure, in terms of writing, I definitely wouldn't try to get published with the top companies if I was serious about writing. And I could complain about the oligarchal media consolidation in the USA, or I can just deal with it and go on.
It's not actually a very difficult or complicated choice to make, and it's exactly the same with google, if you want in, do what it takes to get in. If it's not working, figure out why. Not everyone should get in, although everyone thinks they should, which tends to get pretty annoying, especially when you actually look at some of the sites people think should be ranking, but that's too unpleasant to get into more.
google isn't a black box, it's a bunch of programming written by programmers who think a certain way. It's a bunch of writing, in other words, it can be read like any other book or program, even if you can't see the source code, you can see what it does.
"They offer search not for you the site owner, but for their clients, the searchers..."
I believe that they offer search to create a win win win situation. They fulfill a need for both searchers and website owners. Doing so successfully they create an environment where they can fulfill what they need.
Google creates a way to index and rank sites better than anyone else. This gives us the opportunity to give Google the opportunity to create an index of our websites so that we have the opportunity to be found by searchers at the same time the searches get the opportunity to go to one place to find our websites. This creates an environment so that Google has the opportunity to fulfill what they need.
There is a balance between the 3 parties that MUST be kept.
I guess this is may view on it.
[edited by: arubicus at 6:38 am (utc) on June 6, 2006]
rbacal, publishing is not unlike the search engine world from what I know, there has been massive consolidation in publishing, I believe that something like 5 corporations now control most publishing, just like 3 search engines control most search traffic.
I don't want to take this on a tangent, and I don't have time anyway, but here's the short response.
Regardless of who OWNS the publishing imprint, the people who make decisions on what gets published or not are different, even if the same company ultimately owns a number of imprints. That's a huge difference.
Same with what gets promoted, on book shelves, etc.
Also, people run it, not algos.
There is NO publisher that has even close to the control over what people can find and access on the net, such that google has.
From where I stand, the modern corporate publishing industy might as well be driven by an algo for how predictable and conservative it has become. And keep this in mind, an algo is made by people, it's not a machine. It's just another way for people to make a process happen, people decided how that process would work, what it would look for, and people adjust it to make it turn out the results they want. Just like a top editor reflects downward to subeditors, who learn what he/she will or won't accept. It's really not as different as you think.
google is a publishing enterprise, like a newspaper, except what it publishes is search results, which are more dynamic and fluid than conventional paper based publishing.
I'd say it might be more accurate to think of it this way: getting a top spot in google for a competitive category is not unlike getting a book review in the new york times. Of course there are far more top spots to be had because there are far more search terms, so it's a lot easier to find your way into a search engine top spot than it is to find your way into a publishing top spot.
Sure you can get a review in a local paper or magazine, but it's not worth as much as that nyt review.
It's very unlikely either event will happen without a lot of work and effort on someone's part at some point.
Anyway, this analogy is probably beaten to death now.
But you might give it some thought, if you are currently published and are having trouble with google, its FAR easier to fix that problem than it is to get published. If this isn't happening for you it's time to reevaluate your long and short term methods I think, and not worry so much about whether analogies are correct or not.
I think search will be regulated one day just as many other industries are today.
Not in the U.S., where search results are protected by the First Amendment. (See the SearchKing v. Google case.)
It has been my observation during the past 4-5 years of doing SEO (in general,) that during the months of May, June, and possibly July, Google has been "Out in the woods" so to speak. My guess is that it is a scheduled maintanance. I'm also making an educated guess that search engines have the lowest traffic during the middle of the year (as opposed to the begining and end of the year.)
If I were to give advice, I would say, change nothing, and google will do what it needs to.
This advice is comming from a webmaster that received 90143+128 (Numbers after + are successful hits on "robots.txt" files) hits from the Google crawler during March of 2006 (stats received from awstats.)
And with only smaller adsense covered sites showing up for most searches... a huge boost in advertising revenue whether the advertisers like it or not (hence the %38 percent reported increase in online advertising spending first quarter, I'd bet that a huge chunk of that percentage came from the last month or so)
my 2 cents... Google's $2 Billion... ;) Good luck all.
Regulation can't fix bugs. G's problems are the result of flaws in the implementation of its algos, or the algos themselves.
In my opinion, and regardless of any flaws in Google's algos, the root of the problem is the ability of people in guessing those algos accurately, in conjunction with the sheer increase in number of websites popping up as we speak. During my very brief experience as a webmaster I have found that most questions been asked everywhere across the web is about revealing how Google works and what can people do to hack it in an attempt to gain more PR and outrank competition. Don't get me wrong, I am not saying that following SEO principles should be seen as an offence; I too try to follow those principles as much as I can - it is just that Google's primary objective is to be a search engine that returns results in a 'natural' way, thus been useful for its users. The over-exercising of SEO practises undermines Google's efforts to improve as a search engine and this is where the trouble begins in my opinion. Google then tries to see how its abusers use practises to artificially improve their ranking and fights back by deploying counter measures aiming to penalise those who try and exploit its algos, regardless of whether the culprit is a naughty webmaster of a small website or even that of a large corporation (i.e. BMW de). Google itself doesn't say 'do not use SEO'; in fact they do suggest a set of guidelines that if followed, makes a website more accessible by most SEs, not just Google. But at the same time, Google cannot afford to neglect issues such as link farms, excessive keyword stuffing, heading abuse and many more - as all these techniques influence search results artificially.
On the other hand, most people here would agree that at the moment we can afford to turn our back to MSN search and other SEs, but not Google (despite the fact that MSN is now indexing pages much more faster than Google - at least in my case - otherwise I can't see why anyone would prefer MSN for search over Google!). In other words there is a cat and mouse situation between Google and webmasters. We all hear from time to time things like 'is the X approach penalised by Google?' and promptly go and make appropriate changes on our websites. No one really knows 100% how Google works, as its algos obviously change constantly to accommodate counter measures for 'new hacks' that webmasters have become aware of. In my point of view, those countermeasures that continuously pile up have given, to a large extend, the headaches that Google is suffering from today. Although I try to be optimistic, I don't see that this problem will be fixed from one day to the next - instead it might take a long time.
Going back to the beginning of the thread, I would pick a) (move ahead with new work and just try to forget the dropped pages/content) anytime, because there is no other alternative. Adding new content, amending and improving a website should be the number one priority for webmasters and leave the rest of the problems for the SEs to resolve for themselves. I am sure Google will find its way out of this mess - eventually. Indeed, it is time to stop sharing our misery, although there isn't much we can do apart from building well structured and useful websites. However, sharing our misery can sometimes be helpful;)
it is just that Google's primary objective is to be a search engine that returns results in a 'natural' way, thus been useful for its users. The over-exercising of SEO practises undermines Google's efforts to improve as a search engine and this is where the trouble begins in my opinion.
I could make the argument that it is natural to try to improve one's ranking. (After all, in "real life," people do things to make themselves more attractive to others.) By extension, I could argue that most SEO practices are ethical; it isn't wrong to get people to link to you, or to link to other places, or to create a large number of densely connected web pages. (And the economics of the Internet makes this inexpensive to do.) OTOH, it would be both unethical and illegal to attempt to manipulate rankings by directly altering the contents of an SE's index, or blocking its crawlers from a competitor's sites.
But at the same time, Google cannot afford to neglect issues such as link farms, excessive keyword stuffing, heading abuse and many more - as all these techniques influence search results artificially.
My general feeling is that G should have foreseen that people would attempt to boost rankings once they realized what factors influenced rankings. Early out in the development of PageRank, some of their engineers claimed their algorithms were resistant to such manipulation. No one could create such a massive undetectable link farm, etc. They didn't take into account how easy and cheap it is to create sites and links today. (Or there was a change in their policy that permitted such link farms to be indexed and rank well.)