Forum Moderators: open
Between dynamic and static Web pages, we probably have about 4 million pages online. Prior to May 16th 2003 a great deal of our inventory and content was well indexed in Google. On May 16th hundreds of thousands of pages from our site, just vanished from Google completely. Traffic from Google dropped to less than 15% of what it had been. The overall affect on our system has been quite startling to say the least. We don't spam and we don't do anything that would in any way jeopardize being indexed by search engines. We don't understand what has happened.
We've written to Google asking what is going on and what we can do to fix the problem. I'm sure many of
you are familiar with the standard Google response to calls for help.
What is confusing to me and the merchants that we represent (and our customers for that matter), is that if our content was so popular with Google users for so many years, why has the ‘new’ Google suddenly determined that it is not worth indexing at all? How is it beneficial to Google users to not have access to merchandise and content that can only be found on pages that are now no longer indexed by Google? Although our situation is a bit extreme with hundreds of thousands of pages dropped, from what I have read on this Forum, it is not an entirely unique story.
As the most popular search engine on the Web, Google does have a responsibility to provide the best possible search results to their users. For a long time our content was a part of that index. Our log files show that tens of thousands of Google users viewed our content every month. All we want to know, is how can we get our content back into Google?
-A Proposal
Most large Internet companies seem to start off being reluctant to work with smaller companies on inter-operability issues. AOL and Yahoo! are good examples of how this mind set has transitioned over the past year. As an example, both of these companies now have strategies for dealing with email problems. This is a part of realizing that to deliver value to their customers they need to work with the folks that their customers work with.
At one time AOL was a nightmare to deal with, when it came to sending large quantities of email to their users. I'm not talking about unsolicited spam, just high volume legitimate email (orders, newsletters, shipping confirmations, etc.) being sent from our system to theirs. Their spam filter went nuts and blocked all email from us to AOL. This would go on for several days until we could clear things up with their postmaster and show that it was not spam. This has happened several times over the past few years, until last month. AOL blocked us again, but this time, they gave us documentation on how to send email to their system and not get blocked. We had to install new email software, modify how we sent email as well as get their approval on the final changes. The end result will be that we won't get blocked by AOL again. Their users are happy, AOL is happy and so are we.
Google does not appear to have reached this point of sophistication yet.
Google's customers need to contact them and let them know there is a problem, so that they realize it and can live up to the needs of their customers. Google does not want to be helpful giving definitive answers to problems posed by Webmasters, because of the optimizations issues involved. If they give out too much info on how to get listed, they feel that people will take advantage of the situation. This is just because they misunderstand the issue. At first AOL didn't want to be helpful with email, because they felt that it would be easier for spammers to get past their filters. They have now realized that they need to work with people to get legitimate mail through to their customers. To accomplish this, they have set up a list of requirements that would work with legitimate mailers. And would not work for spammers.
I don't think it would hurt Google at all, to have an automated page where you could type in a URL and it would tell you why it isn't indexed and when it might be. I think it is in their interest for their index to be complete. If they are missing pages due to some technical problem, they need to know about it and fix it. It is very natural, for Google to think that since lots of people are out to increase their rank in Google by any means necessary, that everybody else is out to beat their system as well. That is the initial mind set of most of these large companies. Over time and out of necessity, they realize that not everybody is a bad guy and that they need to cooperate with their users. Without cooperation they simply can't provide the best possible service to their end users.
[edited by: Marcia at 7:07 pm (utc) on July 3, 2003]
[edit reason] Sorry, no email quotes per TOS. [/edit]
If Google were to tell webmasters what was specifically wrong with each page, then webmasters could then systematically determine what the secret sauce at Google was. This type of empirical research could be done quite easily and the Google algorithm could be well known in a short period of time. I believe that they want to protect this algorithm as much as possible to keep the playing field fair.
I too would love to see feedback as to why my pages are ranked as they are, or even what is “wrong” with them. I have given up on this hope, and in turn am relying on the tips and techniques that are well published within this forum.
I had the same thoughts that you did, but I have changed my approach. I know that this does not answer your question, but this is just my opinion why we will never get the type of feedback that you are requesting.
Your comparison is a bit of apples and oranges.
As for Google. They built their multi-billion dollar business on what? Us. The webmasters. Yes, I know the mantra "its a free service". But were would Google be if all the webmasters decided to "noindex, nofollow"? Obviously they wouldn't, but the point is they have as much right to as Google does to "drop with explaination".
I think if they wish to index our pages, that we are providing them with content, and for that contribution to their success, we deserve more rights. It may be a priviledge right now, and those well-indexed wouldn't look the Google-gift horse in the mouth, but others are working feverishly to grab the throne.
Anyone in business knows that, no matter how big you are, you're nothing without customer service. Does google really understand a customer isn't always the guy who lines their wallet?
And if they feel the users of their SERPs are the customers, I wonder how many of them actually think when they search with Google, they are not getting what they used to. Would they start going elsewere?
Google could provide a service like W3C...where you could check your dropped pages for violations of their rules. They need to spend that IPO money on service to those who's backs they were built on. Lets hope thats why they're stalling, so they can plan a great system.
I'm not sure but you may want to edit out that e-mail quote as e-mail quotes are taboo around here. See #9. [webmasterworld.com]
Google's customers need to contact them and let them know there is a problem, so that they realize it and can live up to the needs of their customers.
In my opinion, Google's main customers are advertisers & searchers. Webmasters are really only customers when they're searching or buying AdWords. I think them reminding you that listings in the main index are a free service supports this.
I guess that's a form letter, but it sure sounds like they're saying, "Your car's in the shop buddy; either pay for a rental, or walk like everyone else until it's repaired".
I too have lost perhaps thousands of dollars on this repair job but market's rise & market's fall; it's just the cost of doing business to me. The upside is I think most of us that are taking a beating will bounce back stronger than ever.
Note the issue that vidpro is talking about is not ranking, it is about pages not being indexed at all.
Yes, I caught that he had around 4 million pages online & a couple hundred thousand are missing. I resemble his ratio albeit on a much smaller scale. :o
disappeared by a big search engine. In both cases content that the end-user has specifically requested is not available.
1 - the search engine did not disappear the pages - the pages still exist - they are just not indexed by the search engine, which is perfectly ok for a search engine to decide to index a page or not at any time.
2 - the user did not specifically request the webpage, the user entered a search term and requested results relevant to that search term from all of the web - and the search engine returns what it sees fit.
the analogy with AOL is flawed, certainly.
however, the question of why the pages may have been dropped remains as a useful part of the post - as does the suggestion that google publish better guidelines on how to make your site google-friendly.
the aol/yahoo analogy doesn't work. aol was stopping you from delivering something that your CUSTOMERS had requested from you. you use google to troll for PROSPECTIVE CUSTOMERS. certainly, when people were searching google for the products that you offer, the serps didn't return a 'no matches' page, did they?
hundreds of thousands? would that be 500,000? this thread just seems curious.
Its obvious that for big operations like you its better to rely on relaible traffic sources like PPC /affiliates/Ad partnerships rather than pure organic listings .
I don't think it would hurt Google at all, to have an automated page where you could type in a URL and it would tell you why it isn't indexed and when it might be.
In a certain way I would like to see such a system in place, but I don't agree that telling people why they were banned is a good idea. There are lots of people with lots of domains, who wouldn't mind losing a few in order to learn what spammy tactics will get them banned...and then they'll come back even better, knowing exactly what they can get away with. Perhaps just an automated form that might tell you the status of a domain - indexed, scheduled for indexing, banned, etc. I don't think that would give too much away.
The problem is, there seems to be a great fear among webmasters of being penalized by google, even if you aren't doing anything spammy or don't put much effort into SEO. You drop a few (or in Esmerelda, a few hundred) positions in the SERPs and start wondering what your crime was. IMO, the crucial detail is not why, but whether you've been banned, so if access to that information was made available, I think we'd all be better off.
I think Google has some responsibility to reduce the stress level that they've created.
I think it is in their interest for their index to be complete. If they are missing pages due to some technical problem, they need to know about it and fix it. It is very natural, for Google to think that since lots of people are out to increase their rank in Google by any means necessary, that everybody else is out to beat their system as well. That is the initial mind set of most of these large companies. Over time and out of necessity, they realize that not everybody is a bad guy and that they need to cooperate with their users. Without cooperation they simply can't provide the best possible service to their end users.
I agree. Recognizing that most people aren't out to cheat the system is very important. Google should take an "innocent before proven guilty" approach and give people the benefit of the doubt.
i agree that, if you are a big ecommerce site, then paying for clicks in an area where there is NO other competition would seem like a no-brainer.
maybe you should hire an internet marketing firm. it would seem to me, again, that any business relying on free serps is pretty much pissing in the wind, and hoping not to get wet.
It seems they don't manufacture or sell these products themselves, so I'd want to know more about what this "aggregating" process is before staking out an opinion. It could just be scads of products from affiliate datafeeds dumped into a database.
[edited by: mwexler at 8:42 pm (utc) on July 3, 2003]
Some but not all of these items are listed on other sites.
If these listings are exact copies of those on other sites, its possible they've been weeded out as duplicate content.
Since you've got all those URLs unindexed, its a great time to make them more SE-friendly and take out the query strings.
In short, I don't think you did anything "wrong" or google did anything "wrong", but a system that deals with 3+ billion of anything can't depend too much of human eyes, so the algorithmic chips fell where they may and this time you lost.
HOWEVER, I still think that having one page each for millions of items and expecting it all to be indexed by Google is excessive. These are transient, low-information pages for things like Gumby lunchboxes and vintage calculators. My God, one site's 4 million pages would be a tenth of a percent of Google's database of the entire Internet.
[google.com...]
"A technical glitch on our side may have caused us to 'miss' your site...."
If your not banned completely, there is probably nothing to worry about.
Now, just a quick question; Since the missing content is so valuable to the web community, have you taken steps like PFI for all 4 million pages on the other search engines?
I expect there are many of us out here who agree.
Perhaps The Googlers will eventually get it all together
in my lifetime.
My God, one site's 4 million pages would be a tenth of a percent of Google's database of the entire Internet.
Exactly...and if you aren't happy with 175,000 pages indexed, there are things you can do with link structures and URL rewriting to help increase that number.
You can't expect google to be a personal gateway to your website.
The only error I can see is how you got this far with 0.5M pages indexed.
he had around 4 million pages online & a couple hundred thousand are missing. I resemble his ratio albeit on a much smaller scale. :o
Of the Pages we had online, we had about 460,000 to 500,000 indexed in Google. We are now at anywhere from 140,000 to 195,000. Most of that loss is inventory listing, so it was quite substantial.
natural wrote:
You use google to troll for PROSPECTIVE CUSTOMERS. certainly, when people were searching google for the products that you offer, the serps didn't return a 'no matches' page, did they?
Yes we do, but the nature of the inventory that we carry is such that in many cases we are probably the only source online for the item that the prospective customer is looking for. How is it beneficial to Google to drop a listings that are not available from any othet online source?
gopi wrote:
I understand you position vidpro2 , but why dont you use premium listings ( or increase the existing premium exposure ) .
Its obvious that for big operations like you its better to rely on relaible traffic sources like PPC /affiliates/Ad partnerships rather than pure organic listings.
Our margins are such that it would not be profitable for us to run our business if we had to run 500,000 item specific ad word campaigns, one for each item we are selling. keep in mind that we don't have multiples of most items. We've done extensive testing with Google ad words for general categories of inventory. Yes they do generate traffic, but our analysis shows that the traffic from these campaigns does not buy.
natural wrote:
are vidpro and mwexler the same person?
nope, mwexler is our CTO.
i agree that, if you are a big ecommerce site, then paying for clicks in an area where there is NO other competition would seem like a no-brainer.
As I mentioned above, the numbers just don't work and we've spent a lot of time and money playing with adwords. If it did work, we'd be all over adwords.
if you aren't happy with 175,000 pages indexed, there are things you can do with link structures and URL rewriting to help increase that number.You can't expect google to be a personal gateway to your website.
The point I was trying to make is that most of our inventory is quite unique. It's not available elsewhere, so Google users are not benefiting by Google dropping our listings. A site like Amazon has over 3 million indexed pages in Google and most of that inventory is available at multiple locations on the Web.
Yes we benefit from being listed in Google, but at the same time, our content has benfited Google users for many years and right now, a great deal of that content is gone.
I can understand your frustration, but this is how companies rise & fall...especially .com's that rely too much on Google, or any single source of advertising for that matter.
So you had a good run for a while...maybe now its time to pack it up and move on. I'm sure you realize that google doesn't owe you anything. In this case I'd say its the other way around.
I'd at least try to make some contact with google to try to figure out what happened. If you were/are an active adwords advertiser that might help you get to someone.
But honestly, I don't see how you can take this so hard. Think of what you can do with 175,000 pages. Does every item need its own page, and if so, do each of those pages need to be indexed? Probably not. Hire a good SEO and get those deep pages closer to the front and make the item lists more effective. If a good SEO couldn't get 4 million "quite unique" items well-indexed on 175,000 pages, then we'd all be just as screwed as you.
Welcome to WW vidpro2, nice to have people here, who describe real world problems on large scale systems...
The message I found between the lines in your mail made me think and I wanted to add:
I think you are right, the Internet is a user focused system. Search engines today are more "automated" driven and because of the ridiculous amount of partys involved: worldwide, billions of pages, billions(?) of domains, etc. the major problem of Google is: they are not user driven, they just found the "common taste" of most of the users. I guess they think about that as well, otherwise they would not have bought the weblog system they have.
For your problem: could it be, that the pages that have been dropped from the index had something making them worth to be dropped:
similiar title like "Produkt information about no. 123"?
any other duplicates in content, where 10% of the page is the product detail and the rest is the usual advertising around it?
that would mean you have been kicked by an algorythm not by a person. Because of spam/cloaking Google gets more restrictive every day and the algos change on the way.
If you violated against some rules without knowing, rethink...
regards,
P!
You have 175,000 pages in the index and in the areas that I am interested in, you don't rank very well. I tried some searches for some items that I collect, and you do have pages in the index (I checked by doing a site: search before doing the main search) and you were on the 15th page of the SERPs.
Now when I tried to get more specific to see if I could get better results on the searches, I got no matches on the site: search. These might be on your deeper pages that were not indexed, but they were very general terms that should be in your standard navigation.
Quite simply, your real content is buried way too deep. By the time you get down to the important keywords, you have a bunch of pages that are <PR1.
I noticed that the majority of your individual shops are at around PR2 and they don't even get to the individual items for at least 2 levels of pages after that. Having your most important keywords on pages that are in that great of danger of not getting crawled is where your big problem is.
While PR is not everything when it comes to getting ranked, you really do need those incoming links if you want your deep pages to be crawled on a regular basis. In fact what you really need is some deep links from other sources to your shops from other sites.
The site is not bad (although my thumbnail estimate of what you have online isn't anything close to 4 million items). I don't really know the auction/collectibles market, but it might be providing a very good service.
The site has about 500,000 items listed. The 4 million number is for all pages on the system, content, archives, categories, subcats, merchant specific pages, dynamic pages etc. The inventory pages for the merchandise were well indexed and seem to be where majority of the pages were dropped.
The point I was trying to make is that most of our inventory is quite unique. It's not available elsewhere, so Google users are not benefiting by Google dropping our listings. A site like Amazon has over 3 million indexed pages in Google and most of that inventory is available at multiple locations on the Web.
Quite simply, Amazon is PR9 the last time I checked. You are PR6. You are in a few DMOZ categories, amazon is in dozens or possibly even hundreds. They have millions of deep links to individual items from around the web. These deep links are enough to make sure that each of those deep linked items will be indexed as well as the pages that those items are linked to.
In this case it is your job to go put in the extra effort to get those pages crawled. Google has no way of knowing that they have unique content, but you do. Help google to find that content and you will be doing yourself, google and the users a favor.
i.e. auctions.yourdomain.com , collectables.yourdomain.com, books.yourdomains.com.
Lets your TLD act as the navigation for the sub domains. Google should treat your sub domains as new sites.
You might also rank higher for those sub domains then they currently do in their current category.
Just my 10 cents.
gg
:)
