Forum Moderators: open
Between dynamic and static Web pages, we probably have about 4 million pages online. Prior to May 16th 2003 a great deal of our inventory and content was well indexed in Google. On May 16th hundreds of thousands of pages from our site, just vanished from Google completely. Traffic from Google dropped to less than 15% of what it had been. The overall affect on our system has been quite startling to say the least. We don't spam and we don't do anything that would in any way jeopardize being indexed by search engines. We don't understand what has happened.
We've written to Google asking what is going on and what we can do to fix the problem. I'm sure many of
you are familiar with the standard Google response to calls for help.
What is confusing to me and the merchants that we represent (and our customers for that matter), is that if our content was so popular with Google users for so many years, why has the ‘new’ Google suddenly determined that it is not worth indexing at all? How is it beneficial to Google users to not have access to merchandise and content that can only be found on pages that are now no longer indexed by Google? Although our situation is a bit extreme with hundreds of thousands of pages dropped, from what I have read on this Forum, it is not an entirely unique story.
As the most popular search engine on the Web, Google does have a responsibility to provide the best possible search results to their users. For a long time our content was a part of that index. Our log files show that tens of thousands of Google users viewed our content every month. All we want to know, is how can we get our content back into Google?
-A Proposal
Most large Internet companies seem to start off being reluctant to work with smaller companies on inter-operability issues. AOL and Yahoo! are good examples of how this mind set has transitioned over the past year. As an example, both of these companies now have strategies for dealing with email problems. This is a part of realizing that to deliver value to their customers they need to work with the folks that their customers work with.
At one time AOL was a nightmare to deal with, when it came to sending large quantities of email to their users. I'm not talking about unsolicited spam, just high volume legitimate email (orders, newsletters, shipping confirmations, etc.) being sent from our system to theirs. Their spam filter went nuts and blocked all email from us to AOL. This would go on for several days until we could clear things up with their postmaster and show that it was not spam. This has happened several times over the past few years, until last month. AOL blocked us again, but this time, they gave us documentation on how to send email to their system and not get blocked. We had to install new email software, modify how we sent email as well as get their approval on the final changes. The end result will be that we won't get blocked by AOL again. Their users are happy, AOL is happy and so are we.
Google does not appear to have reached this point of sophistication yet.
Google's customers need to contact them and let them know there is a problem, so that they realize it and can live up to the needs of their customers. Google does not want to be helpful giving definitive answers to problems posed by Webmasters, because of the optimizations issues involved. If they give out too much info on how to get listed, they feel that people will take advantage of the situation. This is just because they misunderstand the issue. At first AOL didn't want to be helpful with email, because they felt that it would be easier for spammers to get past their filters. They have now realized that they need to work with people to get legitimate mail through to their customers. To accomplish this, they have set up a list of requirements that would work with legitimate mailers. And would not work for spammers.
I don't think it would hurt Google at all, to have an automated page where you could type in a URL and it would tell you why it isn't indexed and when it might be. I think it is in their interest for their index to be complete. If they are missing pages due to some technical problem, they need to know about it and fix it. It is very natural, for Google to think that since lots of people are out to increase their rank in Google by any means necessary, that everybody else is out to beat their system as well. That is the initial mind set of most of these large companies. Over time and out of necessity, they realize that not everybody is a bad guy and that they need to cooperate with their users. Without cooperation they simply can't provide the best possible service to their end users.
[edited by: Marcia at 7:07 pm (utc) on July 3, 2003]
[edit reason] Sorry, no email quotes per TOS. [/edit]
Quite simply, Amazon is PR9 the last time I checked. You are PR6. You are in a few DMOZ categories, amazon is in dozens or possibly even hundreds. They have millions of deep links to individual items from around the web. These deep links are enough to make sure that each of those deep linked items will be indexed as well as the pages that those items are linked to.In this case it is your job to go put in the extra effort to get those pages crawled. Google has no way of knowing that they have unique content, but you do. Help google to find that content and you will be doing yourself, google and the users a favor.
Our PR has been fluctuating from 8 to 6 over the past 3 months.... I completely understand what you are saying. The deep links generated by their affiliate programs generate quite a bit more than revenue.
I suppose what we are guilty of, is assuming that the deep crawl along with the extensive indexing we had been experiencing for so long, would just continue.
Getting back to the original post that started this thread, we have a clean design and good content that has worked well for our merchants and our customers for many years. I understand that this is an SEO board and the purpose is to learn techniques to optimize and understand what works for Google, but am I the only one that feels like the tail is wagging the dog?
So many sites and so many online businesses have been affected by this last change. We are lucky to have most of our traffic coming from other sources, but I know that there are other businesses that are much more dependent on Google for traffic and are suffering because of it.
We will work on our Google optimization because we do want those Google users on our system again. Since we need their users more than they need our content, we will accommodate them.
The information shared by you and other members of this board are greatly appreciated and provide significantly more insight into what is going on and what to do about it, than we've been able to obtain from Google. I find it odd that a company that is so dependent on other companies content, provide so few resources to assist their content providers.
Weather: Bleak
Boxes: OK
Network: OK
GoogleGuy: MIA
Coffee Machine: Overload
Algo: KO
Database: OIKY-DOIKY-A-DO-RON-RON-A-DO-RON-RON
Outlook: Unsettled
But if these are little shops in a mall type thing, and individuals have these shops, then Email every last shop owner and explain how to get back links, I think if every shop owner got atleast some back links your situation would inprove drasticly.
Hope that helps
And when you do a search for the one of a kind thing "blue antique special stainless widget" and before you got some pages to this store/mall. And now you get nothing relevant.
I would have to say this is just one more thing pointing at a broken Google, that is trying to repair. In addition it has been 3 months, or close to.
Where is this magical new and Improved Google? Its been promissed where is it? Will it take a year, time to actually build it after you promissed it?
Wheres the new google? We have lost millions of relevant sites, shirly(j/k surely) the new Google will be worth it ;)
And after 3 months it must be any day now that we will see the "New Google" right guys?
it's almost like webmasters and others of us who rely heavily on google expect them to rollout upgrades and fixes from a staging area to the real world, and have it perfect every time.
man, it's taken me a couple of days to figure out what's wrong with a pittly little 200 page website. i would imagine 4+bill pages would take me considerably longer.
i dunno. they're doing the best that they can. we're doing the best that we can. sometimes it gets frustrating, and sometimes it's extremely rewarding.
My suggestion for Google would be to have a system status page, like a hoster does.
Weather: Bleak
Boxes: OK
Network: OK
GoogleGuy: MIA
Coffee Machine: Overload
Algo: KO
Database: OIKY-DOIKY-A-DO-RON-RON-A-DO-RON-RON
Outlook: Unsettled
For me, I have had it.
I am going spam 100%
<snip>
[edited by: ciml at 2:45 pm (utc) on July 4, 2003]
[edit reason] Please respect all users, all the time. [/edit]
So you had a good run for a while...maybe now its time to pack it up and move on. I'm sure you realize that google doesn't owe you anything. In this case I'd say its the other way around.
I believe vidpro2 has made some pretty good points and all comments will come under scrutiny by someone, but to give the advice, "it's time to pack it up and move on.." I would question who is the face behind the mask with a comment like that.
Those of us who have been unscathed by the recent Google roller-coaster ride (which is not many of us) might be the only ones in the wings who are getting a kick out of all this. And personally the problems with my site seem to be getting a lot better for now as far as Google is concerned. But this is a two-tier problem. Webmasters are fighting through the quagmire of disappearing pages. But quickly forgotten, searchers are coming across pages of results that look like 52-card pickup. The only problem is the queens and jacks are being put in the back of the deck.
Will it get better? Of course. But let's just hope Google makes this recession short and sweet. There's no need to pour salt on an already gaping wound.
If you are going to Spam, do it carefully on new, throw away domains. As Spamming, ultimately, has to be a short term project. If you want to get into the top rankings now, you have to Spam to the line. But that line will change, so be ready to throw away those sites and buy / make some more.
I am sure Google will sort it out (hopefully) and that content sites will again prevail. So don't make your content site a Spam, or too closely related to one.
Actually, I am not against what you are doing right now as it happens. I just think you should put it into perspective. And if the SERPS are Spam in your mind (a lot are in mine), then sure. Why not? Spam carefully away.
BTW, I missed one;
Surf: Up
I believe vidpro2 has made some pretty good points and all comments will come under scrutiny by someone, but to give the advice, "it's time to pack it up and move on.." I would question who is the face behind the mask with a comment like that.
OK, you got me, I'm trying to move in on the antiques business. Hehe...really though, no mask here, just being realistic...and remember the big "maybe" in front of that quote.
Yes, I agree with some of the bigger points that have been made here about google providing better feedback (to a point), but don't ask me to pity someone that has 175,000 pages indexed and 0.5M items spread over 4M pages. Clearly a lot can be done to make more efficient use of the number of pages and depth to which googlebot is willing to crawl, as has been pointed out by myself and others.
Maybe it is the cart pulling the horse when an established site has to make changes like that, but sometimes inconvenience is the cost of doing business...and if you aren't willing to pay that price, be prepared to step aside for those who are.
I am sure Google will sort it out (hopefully) and that content sites will again prevail...
GrinninGordon:
I respect your post, I always have.
But I truly think that Google is out on its own agenda now.
I state again: I give in. I will spam like a pro, much like you advise, only 10 fold. The masses will hear from me!
Thank you
But I truly think that Google is out on its own agenda now.
This is new?
Dolemite: So you had a good run for a while...maybe now its time to pack it up and move on.
So, I guess every time the SE's change their algo we should all just pack it up, since we had a "good run for a while!" *cough*
That sounds like a strategy for success if I've ever heard one...
Me thinks Dolemite just wants to move in! ;)
If that was the case, I wouldn't be so worried. However, the serps I've studied show spamsites on top on highly competitive keywords with no backlinks whatsoever.
With a PR of maybe 3 or 4.
Some don't even have the keyword on the page. Not in title, and neither in any backlink.
I could live with spam topping the serps if it was spam that got ranked that way by hard labor. Or by some clever trick. Or by anything else, really.
But as things appear now the only reason this garbage floats on top is the Google variation of 'Our algo's have been drinking - not us' - better yet: there don't appear to be any algo's whatsoever.
Oraqref
Probably because everytime you find something new you come and post it out in the public.
There's an intelligent reason for this forum being called GOOGLE NEWS.
But most of you just don't get it.
Why? Logic would dictate that Google is solving the problems with the core ranking Algo at this time, and can not run the Spam filters as this would cloud their job.
I am pretty sure the whole last two updates were about converting the existing algos from a static, 6 weekly update. To a continuous / rolling / freshbot update. They have admitted they took the data of the time when they first started testing the last update algo, and therefore lost ground with data from the beginning of this year.
I am pretty sure they would NOT have made changes to the main ranking part of the algo while converting it to a rolling algo for the same reasons. This must have been a major jump and job for them, and adding other things into the soup would have been madness.
So, by my book. They are still finishing off the fundamentals of changing from static to rolling algo. And once they have finished, then they will add the Spam filters and catch up with the lost data. Which they are hardly going to admit still exists, are they? Well, at least not directly. The fact GG has not denied this is good evidence enough. The results I have seen are better evidence enough.
Therefore, if you build a Spam site. Don't get upset when they add their filters and it has a gray PR bar (and tools your real content site if you have been dumb enough to link the two).
