Forum Moderators: open
Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.
At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.
The index-page was always in the top-3 position for our main topics, aka keywords.
I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index
As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.
My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.
Lets call them:
www.foo1.net and www.foo2.net
Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)
The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window
Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).
Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.
So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.
Fetched the page, but again got a code 200 with no redirects at all.
Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"
Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.
Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html
Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.
None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo
I have send an email to Google about this and am still waiting for a responds.
But Google's webmaster guidelines are very clearly against cloaking.
The reply I got from webmaster at google.com clearly stated that they did not care about this technique and that it is perfectly within their current guidelines.
Here is the current update on my situation.
Nothing has changed. The offending site was notified, and the reply was that they were not using any type of cloaking, blackhat, etc. They claim it's a bug in google. The link has been removed and that is all they can do,
The SERPs still show their link for my site. Main keyphrases are ranked #3, #5, #50. I tried a suggestion from a past post on checking for a cloak, but could not get it to work correctly (I have a linux box among mostly windows boxes, but am not very proficient).
I have received no response from google, not even a "we have received your email," type of response.
I am kind of a rock in a hard place right now. If the site IS using a cloak, I would like to report it to dmoz, yahoo, and G. If not, and it's a bug in google, I would like to continue to email google and keep this thread alive to push for a change in policy and/or algo.
If there is any one who is proficient at detecting cloaks, or knows and would share this info, I would appreciate a sticky or a response here. I have read past posts and it seems that this may be unintentional by the offending site and I don't want to start flaming/reporting/emailing if that is the case.
Thanks.
P.S. Oh... and to DaveAtIFG, I only received a shoulder wound, it was hard staying off the net this weekend :-)
Because:
"your webmaster must first insert the appropriate meta tags into the page's HTML code"
which are:
<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">
This is very normal, as it would be too easy to have pages removed from sites you dont like.
The redirect URL (link.php?id=blabla) is a meta redirect to *their* homepage, while Google's cache shows webdude's Site.
This is definitely cloaking.
Disguising as Googlebot doesn't help, the cloaking must be IP based.
If you can't get Google to respond to this problem with Google search
See my note above.
Google has responded.
They stated that they do not consider this to be a problem and that I should leave them alone.
[edited by: quotations at 4:48 pm (utc) on Sep. 20, 2004]
vs.
> They stated that they do consider this to be a problem and that I should leave them alone.
Which one is the right one?
Both are now right.
Sorry for the missing word.
webmaster @ google .com believes that they are handling this properly.
They do not care if the top site about widget gardening in the SERPS is actually a porn site because of this bug.
You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.
Seems that this isn't just exclusive to google either. I have read posts on the Y forum of this as well.
But here is the kicker!
I checked the EXACT same keywords in Yahoo, and they all link DIRECTLY to my page. In other words (stay with me now...)
keyword1 keyword2 keyword3 returns #3 in Google, #6 in Yahoo - Yahoo links to my page, Google links to offending site.
keyword1 keyword2 keyword4 returns #5 in Google, #5 in Yahoo - Yahoo links to my page, Google links to offending site.
keyword1 keyword2 returns #50 in Google, #63 in Yahoo - Yahoo links to my page, Google links to offending site.
So now, at least for me, it seems to be related to Google only. However, if you think about it, this may give more weight to the argument for cloaking if the cloak is IP based. Google only? Don't know yet.
I still have 2 subpages of my site in the #500s of G, but nothing for main keyphrases or links to my home page unless you do a search for the title in which I have the first 3 spots now for various pages.
HEY! Wait a minute. A search for the title last week showed my site #1 and the offending site #2. I can't find the offending site now out of the first 100 results.
Is this progress? I'll have to wait and see.
Still awaiting reponses for the cloaking test.
I wish I was more familiar with the technique, I would hijack some really high profile sites (how about google itself?) instantly and see whether cloaking *then* becomes an issue for google.
quotations,
You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.
The Google response ("go pound sand") was actually better than the Yahoo response ("Your site has been manually removed from the index.")
The cloak is easy to spot.
1. Find the offending page in the SERPS.
2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.
3. Click on the SERP link to see the porn gateway page on their site.
4. Complain to Google.
5. Get told that they do not care about content on pages and there is nothing wrong with this practice.
6. Complain to Yahoo.
7. Get told your site has been removed from the index and will not be returned.
2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.
I am not sure if this is the case. It is entirely possible that this is a bug in google and not a cloak. That is the crux of the problem. Cloaking or bug?
Granted, the cache shows the orif\ginal site, but whether a clok is in place is a little bit more difficult to find out. Especially if it is IP based.
Granted, though, regardless if it is a bug or a cloak, the problem should be fixed.
