homepage Welcome to WebmasterWorld Guest from 54.161.240.10
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 [7] 8 9 10 11 12 13 > >     
Dupe content checker - 302's - Page Jacking - Meta Refreshes
You make the call.
Marcello




msg:59535
 11:35 am on Sep 7, 2004 (gmt 0)

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

 

worker




msg:59715
 1:59 am on Sep 19, 2004 (gmt 0)

Thanks Claus!

quotations




msg:59716
 6:26 am on Sep 19, 2004 (gmt 0)

But Google's webmaster guidelines are very clearly against cloaking.

The reply I got from webmaster at google.com clearly stated that they did not care about this technique and that it is perfectly within their current guidelines.

webdude




msg:59717
 1:30 pm on Sep 20, 2004 (gmt 0)

Well, I am back.

Here is the current update on my situation.

Nothing has changed. The offending site was notified, and the reply was that they were not using any type of cloaking, blackhat, etc. They claim it's a bug in google. The link has been removed and that is all they can do,

The SERPs still show their link for my site. Main keyphrases are ranked #3, #5, #50. I tried a suggestion from a past post on checking for a cloak, but could not get it to work correctly (I have a linux box among mostly windows boxes, but am not very proficient).

I have received no response from google, not even a "we have received your email," type of response.

I am kind of a rock in a hard place right now. If the site IS using a cloak, I would like to report it to dmoz, yahoo, and G. If not, and it's a bug in google, I would like to continue to email google and keep this thread alive to push for a change in policy and/or algo.

If there is any one who is proficient at detecting cloaks, or knows and would share this info, I would appreciate a sticky or a response here. I have read past posts and it seems that this may be unintentional by the offending site and I don't want to start flaming/reporting/emailing if that is the case.

Thanks.

P.S. Oh... and to DaveAtIFG, I only received a shoulder wound, it was hard staying off the net this weekend :-)

worker




msg:59718
 3:14 pm on Sep 20, 2004 (gmt 0)

A recent post suggested using the URL removal form in Google. If the link you are referring to is removed from Google, then hopefully Google will find and replace your site's URL in their database soon after.

Marcello




msg:59719
 3:35 pm on Sep 20, 2004 (gmt 0)

The "URL Removal Tool" from Google only works to remove pages from your own site, not to remove pages from someone else's site.

Because:
"your webmaster must first insert the appropriate meta tags into the page's HTML code"
which are:
<META NAME="GOOGLEBOT" CONTENT="NOINDEX, NOFOLLOW">

This is very normal, as it would be too easy to have pages removed from sites you dont like.

dirkz




msg:59720
 4:08 pm on Sep 20, 2004 (gmt 0)

Just took a look at webdudes problem:

The redirect URL (link.php?id=blabla) is a meta redirect to *their* homepage, while Google's cache shows webdude's Site.

This is definitely cloaking.

Disguising as Googlebot doesn't help, the cloaking must be IP based.

questwtg




msg:59721
 4:16 pm on Sep 20, 2004 (gmt 0)

This is very interesting. I had no idea this page/site hijacking is possible in Googles search. If you can't get Google to respond to this problem with Google search. Maybe the news site can get Google to respond to this question of a Google page/site hijacking. Maybe if you all emailed a link of this discussion to your favorite news site, they may publish a news story about this problem. Then Google would have to reposed to the news report.

quotations




msg:59722
 4:39 pm on Sep 20, 2004 (gmt 0)

If you can't get Google to respond to this problem with Google search

See my note above.

Google has responded.

They stated that they do not consider this to be a problem and that I should leave them alone.

[edited by: quotations at 4:48 pm (utc) on Sep. 20, 2004]

dirkz




msg:59723
 4:47 pm on Sep 20, 2004 (gmt 0)

> The reply I got from webmaster at google.com clearly stated that they did not care about this technique and that it is perfectly within their current guidelines.

vs.

> They stated that they do consider this to be a problem and that I should leave them alone.

Which one is the right one?

quotations




msg:59724
 4:51 pm on Sep 20, 2004 (gmt 0)

Do [not] consider it to be a problem.

Both are now right.

Sorry for the missing word.

webmaster @ google .com believes that they are handling this properly.

They do not care if the top site about widget gardening in the SERPS is actually a porn site because of this bug.

webdude




msg:59725
 5:02 pm on Sep 20, 2004 (gmt 0)

quotations,

You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.

Seems that this isn't just exclusive to google either. I have read posts on the Y forum of this as well.

But here is the kicker!

I checked the EXACT same keywords in Yahoo, and they all link DIRECTLY to my page. In other words (stay with me now...)

keyword1 keyword2 keyword3 returns #3 in Google, #6 in Yahoo - Yahoo links to my page, Google links to offending site.

keyword1 keyword2 keyword4 returns #5 in Google, #5 in Yahoo - Yahoo links to my page, Google links to offending site.

keyword1 keyword2 returns #50 in Google, #63 in Yahoo - Yahoo links to my page, Google links to offending site.

So now, at least for me, it seems to be related to Google only. However, if you think about it, this may give more weight to the argument for cloaking if the cloak is IP based. Google only? Don't know yet.

I still have 2 subpages of my site in the #500s of G, but nothing for main keyphrases or links to my home page unless you do a search for the title in which I have the first 3 spots now for various pages.

HEY! Wait a minute. A search for the title last week showed my site #1 and the offending site #2. I can't find the offending site now out of the first 100 results.

Is this progress? I'll have to wait and see.

Still awaiting reponses for the cloaking test.

gemini




msg:59726
 5:50 pm on Sep 20, 2004 (gmt 0)

would
Disallow: /links.php
to remove the cached pages from Google's index do the job for all the /links.php?333 pages? Or there actually has to be a list of all the pages? Thanks

dirkz




msg:59727
 5:59 pm on Sep 20, 2004 (gmt 0)

If a link has page A as content in the Google cache but redirects to page B in the actual search and both are on different domains ... Deception can't go any further.

I wish I was more familiar with the technique, I would hijack some really high profile sites (how about google itself?) instantly and see whether cloaking *then* becomes an issue for google.

quotations




msg:59728
 7:34 pm on Sep 20, 2004 (gmt 0)

quotations,
You actually got a response from them like that? I guess I would have preferred no response as in my case. Of course, I am still unsure of the cloaking issue. I still don't have hard evidence on whether a cloak is happening or not. So, I am still unsure as to if it is a google problem or not.

The Google response ("go pound sand") was actually better than the Yahoo response ("Your site has been manually removed from the index.")

The cloak is easy to spot.

1. Find the offending page in the SERPS.

2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.

3. Click on the SERP link to see the porn gateway page on their site.

4. Complain to Google.

5. Get told that they do not care about content on pages and there is nothing wrong with this practice.

6. Complain to Yahoo.

7. Get told your site has been removed from the index and will not be returned.

webdude




msg:59729
 7:50 pm on Sep 20, 2004 (gmt 0)

2. Click on the Cached Version to see the page which is on your site. That is what Googlebot saw and indexed and ranked the page based on.

I am not sure if this is the case. It is entirely possible that this is a bug in google and not a cloak. That is the crux of the problem. Cloaking or bug?

Granted, the cache shows the orif\ginal site, but whether a clok is in place is a little bit more difficult to find out. Especially if it is IP based.

Granted, though, regardless if it is a bug or a cloak, the problem should be fixed.

dirkz




msg:59730
 6:51 am on Sep 21, 2004 (gmt 0)

> I am not sure if this is the case. It is entirely possible that this is a bug in google and not a cloak. That is the crux of the problem. Cloaking or bug?

When you look at the source of your link, it's a redirect to *their* site, not yours.

So it would be a *huge* bug in Google to interpret a redirect to /index.php on a domain as redirect to an arbitrary page on another domain. This would make Google absolutely unusable :)

The only possible explanation against cloaking would be: It was a link to your site, but they suddenly changed it to their site yesterday *before* I looked at it and Google takes some time to adopt the new version. This can be solved by looking at it in a few days. But I bet in a few days the situation will still be the same, so it's cloaked.

buddhu




msg:59731
 9:54 am on Sep 21, 2004 (gmt 0)

I've been following this with a growing sense of disappointment that GoogleGuy - or Google in any capacity - refuse to address this in a helpful way. It is an issue that will affect users even more than it does website operators. So much for claims that the user experience is the main consideration of major SEs.

It's a shame I can't get links from email or other websites into WebmasterWorld forum threads to work, otherwise I'd be publicising this thread all over - and not just in places frequented by webmasters and techies.

Almost time to vote with the feet and withdraw paid advertising from G and Y!, perhaps.

Patrick Taylor




msg:59732
 11:54 am on Sep 21, 2004 (gmt 0)

I presume the scenario discussed in this thread is repeated many times on the whole of the web - webdude said as much, I think. Of course there will always be scullduggery, web-brigands - call them what you will - exploiting techniques that Google and other search engines publicly state are against their guidelines. No doubt that in the interests of the overall "user experience" - the big picture - these organisations will grind along and eventually fix or improve the situation, though it doesn't quite smack of the jeans and T-shirt jump-into-action "do one thing and do it really really well" Richard Branson style of operation that Google, at least, likes to think of itself as.

The "Google owes nothing to anyone except the surfer" line of argument has had its day, I believe. The web moves on and it's about time they set up a proper means of communication with individual webmasters, and I don't care how many zillion pages there are in their index - it's no excuse.

dirkz




msg:59733
 12:53 pm on Sep 21, 2004 (gmt 0)

> The "Google owes nothing to anyone except the surfer" line of argument has had its day, I believe.

Of course they owe nothing to the webmasters. But in the surfer's interest they should equally level the playing field for webmasters.

There'll always be extremes: Webmaster that'd never violate the guidelines and the ones that will do it on every occasion. But if the undecided majority in the middle sees that it pays off and bears no risk, SERPs won't get better.

webdude




msg:59734
 8:04 pm on Sep 21, 2004 (gmt 0)

An update...

The offending site's webmaster has emailed me several times now, asking if he can help, doesn't understand, does not blackhat, blah, blah, blah.

But after some stickies back and forth with some people who are helping, I decided to try some things to see if this guy is either telling the truth or a really good con artist.

I searched for all the links on his site...

site:http.www.widget.com

I got a LOT of links.

I spent the last hour clicking every link to see where it went. I even found my link in there. And guess what? Out of all the links, mine was the only one that redirected. to his home page. Not sure why, but that is what happened

That doesn't tell me much in itself because my link doesn't exist on his site anymore and a redirect to the home page would be perfectly acceptable.

But I did not find any sites that had a meta refresh going to the correct site. So one of 2 things has happened here. Either my site was the only one that had a meta refresh originally (which I highly doubt) or the webmaster of the offending site got rid of all the meta refreshes. This seems doable. At least that is what I deduced so far. The links were actually going to the offending site (as it should be).

BUT! And this is really BIG but...

Next I ran a lot of the links through a header checker and guess what? All of them show redirects to the correct sites. So I wonder how this is being pulled off? The links in the SERPs go to the offending site now and stop, but the header checker shows a redirect to the original site and then a 200 on that site. What The?

Next I randomly checked the cache for the links in the serps and found almost half of every link shows the original homepage cached exactly like mine.

Next I randomly checked the offending links against the homepages using the link: command. Once again almost half of the backlinks for the offending site are exactly the same as the original homepages.

So what does this all tell me? I have no idea! Whatever is happening appears to be random. The offending links are taking on the cache and the backlinks of the original sites. Why some and not others, I am not really sure.

Could be the guy was caught and he is trying to reverse the process but google is slow on showing the changes in the SERPs. Could be that he has no control over what is happening and the SERPs are just falling where they may.

Anywayz...

Thought I would do an update. I am going to keep this alive 'til I get my site back.

p.s. Sent new emails today to google. No response yet, not even the standard auto-reply.

idoc




msg:59735
 8:12 pm on Sep 21, 2004 (gmt 0)

I think these type operators are caching the original content for the googlebot by i.p. address and the rest of us get the seemingly inane pseudo-directory listings when we click the links. I think google is on it as I see them all over my *borrowed* pages asking for paths I have never had. I had originally written the webmaster at google with 26 domains that were using my content a few months back.

Bluepixel




msg:59736
 9:00 pm on Sep 21, 2004 (gmt 0)

You guys are really stupid, sorry.
It's a bug in google, I experienced it myself because I linked sites with a Location:.. redirect too.
It has nothing to do with the webmaster doing any cloaking.

idoc




msg:59737
 9:54 pm on Sep 21, 2004 (gmt 0)

As some have said it *can* be innocent... i think that is also right sometimes though the flaw is also being manipulated in black hat seo i believe having traced ip's back to various seo and marketing folks.

MikeNoLastName




msg:59738
 12:39 am on Sep 22, 2004 (gmt 0)

>Could be the guy was caught and he is trying to reverse the process but google is slow on showing the changes in the SERPs.

It could just be me, since we've been doing a lot of moving, but has anyone else noticed if the specific pages apparently being "hijacked" are ones they've recently redirected themselves or made major changes to (like title or major keywords)?

All the ones I've noticed it on, in our case, are only ones which we recently 301'd and or title changed.

I have a theory that they may have been around far longer than we knew, but were being highly penalized in the SERPs, but are only now "bobbing" to the surface of the SERPs as we change the originals.

kaled




msg:59739
 12:42 am on Sep 22, 2004 (gmt 0)

It has nothing to do with the webmaster doing any cloaking.

Cloaking is not required but it is possible that it is being used to cover tracks - that's all.

Here's a scary thought - one for the chainsaw.
What happens if, in addition to a meta redirect, you add a robots noindex meta?

I'd say it's an even money bet that the target page will be nuked without leaving any explanation in the Google cache.

Kaled.

DaveAtIFG




msg:59740
 6:44 am on Sep 22, 2004 (gmt 0)

I stuck a couple test pages up tonight to see how Google reacts to a 301, a 302, and a meta refresh. These are on an old and stable site. A header check on the page with the meta refresh returns a 302 response.

Googlebot visits this site about every 4-5 days, last seen 9/19, so it will be a few days before I know anything. I also stickymailed webdude with specifics so he can follow along and confirm things.

Kerrin




msg:59741
 7:49 am on Sep 22, 2004 (gmt 0)

My question from post 106:
Googlebot only seems to treat a Meta Refresh like a 302 when a page redirects to an external page.
Eg: site1.com/bla.html => site2.com/blabla.html

Has anyone seen an internal Meta Refresh acting like a 302?
Eg: site1.com/bla.html => site1.com/blabla.html

DaveAtIFG, could you try both internal and external meta refreshes? It is my belief that they are treated differently.

webdude




msg:59742
 1:30 pm on Sep 22, 2004 (gmt 0)

You guys are really stupid, sorry.
It's a bug in google, I experienced it myself because I linked sites with a Location:.. redirect too.
It has nothing to do with the webmaster doing any cloaking.
I am beginning to agree with this too. Not the stupid part. However, I believe there might be other factors involved. I can't seem to put my finger on it. So far on the sites I have found that are exhibiting the cache and backlinks problem seem to be random.

As some have said it *can* be innocent... i think that is also right sometimes though the flaw is also being manipulated in black hat seo i believe having traced ip's back to various seo and marketing folks.
I agree with this too. It can be innocent, especially if it is a bug. But this is what I am trying to find out in my case. I don't want to flame a site that is innocent. So how do you find out? I have had some check and have gotten differing views no this.

I even found s cloak checker on the web, but it seemed cheesy and did not show what it found. It just basically said "yes" or "no" on the cloaking. It's hard to put any faith in that.

MikeNoLastName wrote...
It could just be me, since we've been doing a lot of moving, but has anyone else noticed if the specific pages apparently being "hijacked" are ones they've recently redirected themselves or made major changes to (like title or major keywords)?
Now, I find this extremely interesting because of 2 points here. Yes, I changed the title of my homepage about 2 and a half months ago from "Name of Site - Dedicated to Location Widget" to Location Widget at Name of Site" Could there be a correlation here!

Also, I renamed some of the filenemes on the site. I added 301s to all the renamed files as per other threads and advice. I thought nothing of this at the time. I knew it would take a while for the SERPs to straighten out. BUT, I posted in other threads of a problem I thought I was having with googlebot and the 301s. It appeared that the problem was two-fold. First, every time googlebot started a deep crawl, it would get to one of these redirects and then just stop. No more crawling of any other pages. It would then repeat the process the next day, crawl until it hit a 301, then stop. Next problem was that googlebot, in random fashion, would keep coming back and try to crawl the 301s. Just the 301s and nothing else. It was as if googlebot was having problems with the redirect. I checked the redirects with a header checker and it showed the 301 was being directed correctly, just googlebot couldn't seem to follow them.

If this is related in some way as to why this offending site has replaced my link in the SERPs and shows the exact same backlinks and a cache snapshot of my homepage, I am at a total loss. I would just like to get this mess straightened out.

I stuck a couple test pages up tonight to see how Google reacts to a 301, a 302, and a meta refresh. These are on an old and stable site. A header check on the page with the meta refresh returns a 302 response.
Thanks DaveAtIFG. This is going to be interesting and maybe we can get to the bottom of this.

worker




msg:59743
 2:51 pm on Sep 22, 2004 (gmt 0)

A client I consult for has had their company URL replaced in the G SERPS by an affiliate partner URL. Needless to say, they are unhappy that everytime someone finds them in Google, the affiliate partner is getting credit for that person.

The affiliate is running their link to the client through a click-management script, and the affiliate site has a VERY high 'rank' within G. At this point, my client is going to end the relationship with the affiliate if they do not change the way they are linking, even though they are a KEY partner.

They made this decision because of the last email reply from G on this issue, basically saying that both URL's (the client URL, and the affiliate URL) have the same content, so G is listing the one with the higher 'rank'.

I have to admit to being blown away by the reply from G, and I'm hoping it was from a low-level customer support person who was just wrong...although it would explain the existing behavior of G on this issue.

In this case the client's URL has been hijacked by another site with a higher 'rank', and G acknowledged it...and even explained it with a comment about the 'same content'.

They didn't address the fact that there is really only one page/site of content and that the affiliate is just re-directing.

I can't believe this is the way they will leave things, because it just doesn't make sense.

Maia




msg:59744
 6:49 pm on Sep 22, 2004 (gmt 0)

I also changed the title of my index page, and put 301 redirects on several pages before this happened.

I found a cloaking checker that does give you the html of the pages it's looking at. When I used "Googlebot" as the user agent the only differences I could find was that Googlebot doesn't see one of two cookies, and that is the tracking cookie. Of course, I don't know if Google doesn't see any tracking cookies, or just in this case. Also, Googlebot doesn't get the mySQL database error. Anybody?

I checked lots of the sites that this directory links to earlier, but I could never find any other instance of hijackings. Just my site.

Today, the other site still shows up in Google when searching for my domain name. I did try the Google URL remover, but it said it couldn't remove it because it's a live link.

Since the site removed the link when I asked them to, and the link and cached pages of my site disappeared shortly thereafter, I can't see filing a DMCA complaint. There is no cache when searching for a domain name, and when you try to get any page that doesn't exist on the server of the directory that linked to me, you get the same page that is showing up when you search my domain name. It seems to me my problem is now solely with Google still showing that site instead of mine when searching for my domain name, and I don't know how to make it go away since Google won't help.

webdude




msg:59745
 7:18 pm on Sep 22, 2004 (gmt 0)

Maia,

I am sort of in the same boat. My site was alos remove, but the link remains and is redirected to the offending sites home page.

As for the cloak checker, most of the sites utilizing cloaks are using ip based cloaks which a tool cannot check. You would have to spoof the IP of googlebot in order to do this. A good cloak is transparent to the user and very hard to find. This is what is making this so frustrating.

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 6 [7] 8 9 10 11 12 13 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved