Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.
Seems that this isn't just exclusive to google either. I have read posts on the Y forum of this as well.
But here is the kicker!
I checked the EXACT same keywords in Yahoo, and they all link DIRECTLY to my page. In other words (stay with me now...)
keyword1 keyword2 keyword3 returns #3 in Google, #6 in Yahoo - Yahoo links to my page, Google links to offending site.
keyword1 keyword2 keyword4 returns #5 in Google, #5 in Yahoo - Yahoo links to my page, Google links to offending site.
keyword1 keyword2 returns #50 in Google, #63 in Yahoo - Yahoo links to my page, Google links to offending site.
So now, at least for me, it seems to be related to Google only. However, if you think about it, this may give more weight to the argument for cloaking if the cloak is IP based. Google only? Don't know yet.
I still have 2 subpages of my site in the #500s of G, but nothing for main keyphrases or links to my home page unless you do a search for the title in which I have the first 3 spots now for various pages.
HEY! Wait a minute. A search for the title last week showed my site #1 and the offending site #2. I can't find the offending site now out of the first 100 results.
Is this progress? I'll have to wait and see.
Still awaiting reponses for the cloaking test.
I wish I was more familiar with the technique, I would hijack some really high profile sites (how about google itself?) instantly and see whether cloaking *then* becomes an issue for google.
quotations,
You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.
The Google response ("go pound sand") was actually better than the Yahoo response ("Your site has been manually removed from the index.")
The cloak is easy to spot.
1. Find the offending page in the SERPS.
2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.
3. Click on the SERP link to see the porn gateway page on their site.
4. Complain to Google.
5. Get told that they do not care about content on pages and there is nothing wrong with this practice.
6. Complain to Yahoo.
7. Get told your site has been removed from the index and will not be returned.
2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.
I am not sure if this is the case. It is entirely possible that this is a bug in google and not a cloak. That is the crux of the problem. Cloaking or bug?
Granted, the cache shows the orif\ginal site, but whether a clok is in place is a little bit more difficult to find out. Especially if it is IP based.
Granted, though, regardless if it is a bug or a cloak, the problem should be fixed.
When you look at the source of your link, it's a redirect to *their* site, not yours.
So it would be a *huge* bug in Google to interpret a redirect to /index.php on a domain as redirect to an arbitrary page on another domain. This would make Google absolutely unusable :)
The only possible explanation against cloaking would be: It was a link to your site, but they suddenly changed it to their site yesterday *before* I looked at it and Google takes some time to adopt the new version. This can be solved by looking at it in a few days. But I bet in a few days the situation will still be the same, so it's cloaked.
It's a shame I can't get links from email or other websites into WebmasterWorld forum threads to work, otherwise I'd be publicising this thread all over - and not just in places frequented by webmasters and techies.
Almost time to vote with the feet and withdraw paid advertising from G and Y!, perhaps.
The "Google owes nothing to anyone except the surfer" line of argument has had its day, I believe. The web moves on and it's about time they set up a proper means of communication with individual webmasters, and I don't care how many zillion pages there are in their index - it's no excuse.
Of course they owe nothing to the webmasters. But in the surfer's interest they should equally level the playing field for webmasters.
There'll always be extremes: Webmaster that'd never violate the guidelines and the ones that will do it on every occasion. But if the undecided majority in the middle sees that it pays off and bears no risk, SERPs won't get better.
The offending site's webmaster has emailed me several times now, asking if he can help, doesn't understand, does not blackhat, blah, blah, blah.
But after some stickies back and forth with some people who are helping, I decided to try some things to see if this guy is either telling the truth or a really good con artist.
I searched for all the links on his site...
site:http.www.widget.com
I got a LOT of links.
I spent the last hour clicking every link to see where it went. I even found my link in there. And guess what? Out of all the links, mine was the only one that redirected. to his home page. Not sure why, but that is what happened
That doesn't tell me much in itself because my link doesn't exist on his site anymore and a redirect to the home page would be perfectly acceptable.
But I did not find any sites that had a meta refresh going to the correct site. So one of 2 things has happened here. Either my site was the only one that had a meta refresh originally (which I highly doubt) or the webmaster of the offending site got rid of all the meta refreshes. This seems doable. At least that is what I deduced so far. The links were actually going to the offending site (as it should be).
BUT! And this is really BIG but...
Next I ran a lot of the links through a header checker and guess what? All of them show redirects to the correct sites. So I wonder how this is being pulled off? The links in the SERPs go to the offending site now and stop, but the header checker shows a redirect to the original site and then a 200 on that site. What The?
Next I randomly checked the cache for the links in the serps and found almost half of every link shows the original homepage cached exactly like mine.
Next I randomly checked the offending links against the homepages using the link: command. Once again almost half of the backlinks for the offending site are exactly the same as the original homepages.
So what does this all tell me? I have no idea! Whatever is happening appears to be random. The offending links are taking on the cache and the backlinks of the original sites. Why some and not others, I am not really sure.
Could be the guy was caught and he is trying to reverse the process but google is slow on showing the changes in the SERPs. Could be that he has no control over what is happening and the SERPs are just falling where they may.
Anywayz...
Thought I would do an update. I am going to keep this alive 'til I get my site back.
p.s. Sent new emails today to google. No response yet, not even the standard auto-reply.
