Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
to get a huge load of pages listed, all constructed in the same way, such as:
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
Do you forsee the issue to be solved real soon?
The problem has been there for months already. There were no response or any assurance from google or the other search engines that the issue will be addressed.
The hijackers are thriving on it and may be enjoying the harvest through the coming holiday seasons. They are hoping everyone will be sitting on it and not do anything.
I know it is not the solution but is there any other way those affected can do instead of waiting? Can someone at google respond?
I have only afew pages affected thus not too concerned but I know that the scope that these hijackers hit at are quite broad.
and as useless as trying to take Google to court over something that is most likely being worked on in any case
Taking Google to court is probably not the right action, however, this idea that the problem is being worked on is one I just don't get.
When a security flaw is found in Windows, MS can get a fix out in days (not always, but it does happen). This problem has been known for months, possibly more than a year and still no fix is in sight, nor have Google even conceded that the problem exists.
The possible explanations for this tardiness are as follows :-
1) Google just doesn't care.
2) The code is incomprehensible and the guy that wrote it has left/died, etc.
3) The code is fine but no one is smart enough to understand it.
4) The source code has been lost and no one is smart enough to work out how to patch the binary code.
Of course, the last possibility is interesting. They would probably need someone in his/her fifties or sixties to patch binary code - it's certainly not a skill taught at university.
In the future, Google is unlikely to comment on this issue/problem(?)/bug(?) (or any other), since any "official" comment could be used in support of legal action, whether frivolous or justified.
In my experience, Kerrin's canned response saying "we'll pass this on to our engineers" (in message 41) is an indication that Google is taking the problem seriously and exploring resolutions.
Google needs to first decide if this is their problem or a shortcoming in the HTTP protocol specs that everyone is expected to conform to. Then they must decide if this is a spider problem or an indexing problem. Then they need to fix the appropriate code and test the fix thoroughly before deploying it.
If it's a spidering problem, it likely means the improperly redirected links will need to be respidered, then reindexed. An indexing problem COULD be fixed more quickly if indexing is done independently from spidering I suppose. Google may have a fix in place now and we may be simply waiting for respidering/reindexing corrections to percolate into the publicly available SERPs.
Each of you needs to report your experience with this problem to Google. This page [google.com] suggests reporting it to firstname.lastname@example.org. They need solid examples of the problem to analyze so they can implement a fix. Reporting it or complaining about it at WebmasterWorld does little to improve the situation, although it is an opportunity to vent and commiserate with others experiencing the problem... ;)
www.widgets.com - has text on homepage describing widgets and stuff. In particular there is a line: "Widgets are the new gudgeon clips."
Searching on google for that line should only find my widgets site but also listed is:
The google entry for foo shows my text description but clicking the link goes to the foo site which shows nothing about widgets or gudgeon clips at all.
Is my site being hijacked like marcello's? How can I probe foo.com to see if there is a meta redirect?
From what I gather, when a site is hijacked then google will drop the site and PR0 it.
If that is the case then how quickly does that happen?
Is my site in the early stage of being dropped by google?
Digital Millennium Copyright Act
It is our policy to respond to notices of alleged infringement that comply with the Digital Millennium Copyright Act (the text of which can be found at the U.S. Copyright Office Web Site, [lcWeb.loc.gov ]) and other applicable intellectual property laws, which may include removing or disabling access to material claimed to be the subject of infringing activity. If we remove or disable access to comply with the Digital Millennium Copyright Act, we will make a good-faith attempt to contact the owner or administrator of each affected site so that they may make a counter notification pursuant to sections 512(g)(2) and (3) of that Act. It is our policy to document all notices of alleged infringement on which we act. A copy of the notice will be sent to a third party who will make it available to the public.
Google does not have any kind of duty or obligation to index anybody's website. If they decide they only want to show results for domains that begin with the letter 'A' and were registered on a Tuesday, that's their right. If you don't like how they operate, make your own search engine.
That said, I think this is a pretty major flaw in their algorithm and I, for one, would like to see more extensive analyses of the problem and how it could be resolved. And I'd really be interested in seeing something from GoogleGuy explaining why they do things this way. Enough whining about lawsuits, let's get back to discussion of the actual problem itself.
lynx -mime_header HIJACKURL
where HIJACKURL is the url of the potential hijacker, including "http..." Lynx will show you the HTTP headers, so you can tell if it's a 301, 302, or meta refresh.
Search Google for "wannabrowser".
under "Agent Selection" use "NetSpider"
UN-Check "Follow Redirects"
Leave "Show HTTP Response Headers" Checked
Yesterday I have send a DMCA-complaint to Google and to Altavista by Fax concerning this matter ... hopefully I will get an answer to this fax.
Today all my more than 8,000 backlinks to my site "www.widget.com" have disappeared.
entering link:www.widget.com now gives as answer:
"Your search - link:www.widget.com - did not match any documents."
So now the hijacking-page must have been completely accepted as "THE PAGE" and as Frank_Rizzo says in message 81, the next step will be that my PR6 will become PR0, resulting in a complete loss of:
- a 4-year old site
- over 80,000 pages
- PR6 ranking
- over 30,000 uniques/day
- 200,000 pageviews/day
All of the above the result of someone adding the following line of code to a not so high-ranking page:
"<meta http-equiv="refresh" content="0; url=http://www.widget.com/">"
Also Today (its morning here) My pages in the Google-Index have now dropped from over 80,000 pages to less than 40,000 pages. (using site:www.widget.com)
Also Google traffic is 50% less than the normal average from the last 6-months
I am watching Yahoo like a hawk as I am still getting a lot of traffic from them, but the hijacking-page is STILL NOT not in the Yahoo-Index and my www.widget.com page is still ranking No.1 for its main topic (keywords) on over 3-million results returned.
I still believe in Google and agree with "DaveAtIFG message 79" .... I just hope Google knows about the problem so that other webmasters never have this scenario happen to them.