Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
The main case against Google would probably have to rest on negligence on their part in not fixing the problem after it was brought to light.
I'm not a lawyer but I think a case of negligence would be hard to bring against Google without the existence of a contract of some sort - or at least a reasonable expectation of something by one party from another, or a duty of care or some such thing - and in this case there isn't much.
I think a case of negligence would be hard to bring against Google without the existence of a contract of some sort
Get a clever lawyer and he'll find a case in anything. It is obviously not the case that you need to have a contract with someone before bringing suit. That would bring criminal justice to a complete standstill. How many burglars, murderers or fraudsters get their victims to sign contracts first? ;)
Seriously, just because you're claiming "negligence" does not make a contract mandatory. I can think of numerous recent examples of successful action where there was no contract in place.
Google doesn't owe any of us a living, Google does not owe any of us traffic or ranking. Should they choose to remove us from their listings that is entirely their prerogative. And you can't sue them for that. However, as with any responsible business G will appreciate that they don't need to intentially commit a breach of the law to be named as party in a copyright infringement or in a "collusion" accusation. But I don't see that as the quickest or best route to a resolution.
On a related matter, when "criminal" was mentioned earlier on in the thread it referred to siteowners' deliberate expoitation of this Google flaw and not to Google's activities themselves.
Get a throw away domain, redirect to the page that they hijack from you?
So that their traffic will come to you. Is this workable?
This is about spammers that have found a new way to pull traffic to their site by using 302 and Meta-Redirects at the mercy of popular and well-established websites.
In the area of programming, the Internet and in any area of Software making, there will always be people that will put all their energy in trying to reverse-engineer or burglar into the system.
And the Flaw is not only with Google .... as of Today the same situation has happened in the Altavista SERPS where also since some hours ago my Index-Page at:
www.widget.com
has been replaced by:
www.foo1.com/mykeyword/www-widget-com.html
I am still OK in Yahoo, Jeeves and Alltheweb
Thanks to all for your concerns and lets try to put the hands together to to point this threat and Redirect-Flaw to all Search Engine engineers and more in the news ... not about LawSuits but to warn other webmasters who maybe do not understand why SERPS are dropping their pages and why they are loosing their good ranking they had before.
Thanks for keeping this threath on topic.
If you cannot beat them, can you join them?
That's the worse possible thing to do. More damage caused to the web experience, which damages everybody's sites in a small way and some people's in a bigger way - maybe yours even. Google and other search engines will be aware of the issue raised in this thread and the other related ones, and probably it won't be fixed in a week, but the big picture is that something will be done about it. This maybe won't help the people here whose sites have been affected, at least not quickly enough, but to respond by resorting to the same cynical (or negligent) methods is wrong in principle, and as useless as trying to take Google to court over something that is most likely being worked on in any case.
Do you forsee the issue to be solved real soon?
The problem has been there for months already. There were no response or any assurance from google or the other search engines that the issue will be addressed.
The hijackers are thriving on it and may be enjoying the harvest through the coming holiday seasons. They are hoping everyone will be sitting on it and not do anything.
I know it is not the solution but is there any other way those affected can do instead of waiting? Can someone at google respond?
I have only afew pages affected thus not too concerned but I know that the scope that these hijackers hit at are quite broad.
and as useless as trying to take Google to court over something that is most likely being worked on in any case
Taking Google to court is probably not the right action, however, this idea that the problem is being worked on is one I just don't get.
When a security flaw is found in Windows, MS can get a fix out in days (not always, but it does happen). This problem has been known for months, possibly more than a year and still no fix is in sight, nor have Google even conceded that the problem exists.
The possible explanations for this tardiness are as follows :-
1) Google just doesn't care.
2) The code is incomprehensible and the guy that wrote it has left/died, etc.
3) The code is fine but no one is smart enough to understand it.
4) The source code has been lost and no one is smart enough to work out how to patch the binary code.
Of course, the last possibility is interesting. They would probably need someone in his/her fifties or sixties to patch binary code - it's certainly not a skill taught at university.
Kaled.
In the future, Google is unlikely to comment on this issue/problem(?)/bug(?) (or any other), since any "official" comment could be used in support of legal action, whether frivolous or justified.
In my experience, Kerrin's canned response saying "we'll pass this on to our engineers" (in message 41) is an indication that Google is taking the problem seriously and exploring resolutions.
Google needs to first decide if this is their problem or a shortcoming in the HTTP protocol specs that everyone is expected to conform to. Then they must decide if this is a spider problem or an indexing problem. Then they need to fix the appropriate code and test the fix thoroughly before deploying it.
If it's a spidering problem, it likely means the improperly redirected links will need to be respidered, then reindexed. An indexing problem COULD be fixed more quickly if indexing is done independently from spidering I suppose. Google may have a fix in place now and we may be simply waiting for respidering/reindexing corrections to percolate into the publicly available SERPs.
Each of you needs to report your experience with this problem to Google. This page [google.com] suggests reporting it to webmaster@google.com. They need solid examples of the problem to analyze so they can implement a fix. Reporting it or complaining about it at WebmasterWorld does little to improve the situation, although it is an opportunity to vent and commiserate with others experiencing the problem... ;)
wink wink nudge nudge. . .
