homepage Welcome to WebmasterWorld Guest from 54.147.196.159
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 [6] 7 8 9 10 11 12 13 > >     
Dupe content checker - 302's - Page Jacking - Meta Refreshes
You make the call.
Marcello

10+ Year Member



 
Msg#: 25638 posted 11:35 am on Sep 7, 2004 (gmt 0)

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

 

Marcello

10+ Year Member



 
Msg#: 25638 posted 4:10 am on Sep 17, 2004 (gmt 0)

While I was sleeping......

Woke up this morning to find an answer from Google to my DCMA complaint about the duplicate copy of my Index-Page at foo.com.

The answer comes down to:
Google has removed "the page in question" from the Google-Cache and the archived information for this page ... but searches for widget will still return this page's title and URL until our bot has revisit this page again.

On the demand why "My Index-Page" was banned and replaced by "My Index-Page under foo's URL", the answer is that individual responses are not given, followed by the canned text that eventual penalizations can have many causes, followed by the list of "What Not To Do" as stated in the Google guidelines.
(for which I dont worry as my site is clean)

Furthermore, due to my DCMA complaint to other Mayor Search-Engines, the complete site of "foo.com" is since this morning (my local time) gone from their SERPS and "My Index-Page" under "My URL" has reappeared in its former glory-position.

Things are starting to move .....

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 4:29 am on Sep 17, 2004 (gmt 0)

What this entire post comes down to is greed.

You could not be further from the truth if you tried. For me, there is no money involved. The site that I keep referring to in my posts is a hobby site. Unfortunately, when Brett took his chainsaw to this thread, he deleted one of my posts. But I will repeat what I have stated before...

I don't have any competition. This site does not sell products and I receive no monies from any party. All development is done on my own time at no cost to anyone but myself. I just had an urge to start something that I thought would be kind of cool.

So where in the heck is the greed in that?

And, by the way, no I will not stoop to the black hat measures of others to get even. The black hat sites eventually will go down. Past experience has shown that. And I am not going to get caught in that trap. As stated before, google will eventually fix this. I do have some faith. It just takes time.

Besides, I like to sleep good at night.

Greed huh?

dirkz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 7:53 am on Sep 17, 2004 (gmt 0)

So DMCA complains work to get rid of redirect spammers stealing your hits?

quotations

10+ Year Member



 
Msg#: 25638 posted 9:30 am on Sep 17, 2004 (gmt 0)

So DMCA complains work to get rid of redirect spammers stealing your hits?

It might work on Google but has not so far for me.

On Yahoo, they did a manual permanent ban on my site because I filed the complaint and then wrote to tell me they had done it.

kaled

WebmasterWorld Senior Member kaled us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 12:03 pm on Sep 17, 2004 (gmt 0)

On Yahoo, they did a manual permanent ban on my site because I filed the complaint and then wrote to tell me they had done it.

That is the perfect example of why a code of conduct is required for search engines, together with an independent complaints body. This example also suggests that legislation is required to back it up.

If that happened to me, I would do my best to kick up a big stink on principle.

Kaled.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 12:11 pm on Sep 17, 2004 (gmt 0)

After reading more threads from other forums and some articles on the meta refresh and 302 problem, and because of the way this situation has played out when it comes to my site, I think some of you may be right. There seems to be some sort of cloaking or other tricks going on.

So this brings up another question.

Is there a way to check if a site is cloaking? I read somewhere that you can use firefox, but it is not very effective. I would love to be able to see what googlebot sees when crawling these offending links.

If anyone has any suggestions or knows how this could be done, I would appreciate it. I already tried some spidering tools, different header and html checkers, but the results are pretty much the same. Some of the spider tools just crawl the post 302 or meta refresh link, some tell me they are being blocked.

Marcello glad you are getting some results for your efforts. Keep us informed.

?

dirkz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:26 pm on Sep 17, 2004 (gmt 0)

> Is there a way to check if a site is cloaking?

Look at the google cache, sometimes it's obvious. You can also try to disguise as Googlebot and access the site, but this only works for user-agent based cloaking.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:33 pm on Sep 17, 2004 (gmt 0)

I am looking at the cache. The cache for the meta refresh links show my home page. Is that proof that it is being cloaked? Or does that mean that googlebot crawls the first link to the second link and assumes it's the correct page.

As for the 302s, they show my homepage as well when I view cache, even though the 302 is pointed to the offending sites home page. That is what got me going on the cloak thing in the first place.

my2cents

10+ Year Member



 
Msg#: 25638 posted 2:46 pm on Sep 17, 2004 (gmt 0)

A question and a request.

Request:
Can someone sticky me with an example of a site (any site will do) that is currently being hijacked? I have a friend that is very interested in this topic and Iím trying to gather as much information for her as possible. She is an attorney and believes there may be something here.

Question:
If the date of the file determines the age and ranking of the page when being hijacked, couldnít you simply adjust your date on that page (file) to be older than the hijackerís page?

Thanks!

dirkz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 3:21 pm on Sep 17, 2004 (gmt 0)

I clarify:
Cache says it's your page, but when you click on the search result link and get redirected somewhere else (not your page) than it's cloaked.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 3:29 pm on Sep 17, 2004 (gmt 0)

Thanks dirkz.

Is there any way to see the cloaked page?

dirkz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 3:40 pm on Sep 17, 2004 (gmt 0)

> Is there any way to see the cloaked page?

curl -i -A "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" [widget.com...]

(if you have a Unix compatible box and the cloaking is UA based)

Otherwise, you can fiddle around with the UA of your favorite browser (if it's not IE).

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 3:46 pm on Sep 17, 2004 (gmt 0)

I'll give it a try.

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 4:43 pm on Sep 17, 2004 (gmt 0)

You are unlikely to see anything webdude.

There are two basic approaches to cloaking, one is user-agent based and easily circumvented, the second is IP based and is extremely difficult to penetrate. I'm confident you would need to visit this hijacker site using a googlebot IP to penetrate the cloak. (High quality cloaks often rely on BOTH IP and user-agent checks.)

Google's cache of a page displays exactly what googlebot saw when they spidered the site. If a hijacked site appears in Google's cache, rest assured that cloaking is involved in this hijack. When googlebot visited, probably identified by IP address and possibly confirmed by user-agent, it was served your URL.

Typically, a site using cloaking to serve their own pages (not a hijacker, a simple cloaker) will include a "nocache" metatag on their cloaked pages, to avoid having those pages cached and possibly reveal their cloaking. Since a hijacker is hijacking/cloaking your site, the hijacker cannot add a nocache metatag to your site without hacking into it.

Page jacking has been going on for many years, this is simply a more advanced technique. Is it Google's responsibility to fix it? Not necessarily. But Google's webmaster guidelines are very clearly against cloaking.

There have been similar reports in the Yahoo forum suggesting redirects cause problems. It reportedly effects some sites and not others. I suspect many of these reports are also related to this page jacking trick. And Yahoo webmaster guidelines mirror Google's regarding cloaking.

As Marcia suggested in #121, "There has to be a reason and a reward for doing something like this," so high traffic sites will be the most likely targets and that probably explains why some sites are effected and not others.

Maia

10+ Year Member



 
Msg#: 25638 posted 4:57 pm on Sep 17, 2004 (gmt 0)

OK, so you are saying there is no way my site would have shown up in the cache from another site simply using a meta-refresh and/or 302 redirect to my site?

Because, my site did appear in the cache, but the page was redirecting to my index page. Once they removed the link to me, my page still appeared in the cache, but the page was cached before the link was removed.

Patrick, if you are still following this at all, did you check the cache on the pages you unintentionally hijacked?

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 5:19 pm on Sep 17, 2004 (gmt 0)

As Marcia suggested in #121, "There has to be a reason and a reward for doing something like this," so high traffic sites will be the most likely targets and that probably explains why some sites are effected and not others.

I understand now the difficulty. The offending site is cloaking according to your definition.

The reason and reward? I have thought a lot about that. Right now the reward seems to be that they have sapped my ranking in the SERPs. They have switched the meta refresh that was pointed to my site, to a 302 that is now pointed to ther home page.

One of the key phrases I HAD is now #3 with the link being directed to their home page. My title, mousever description, my homepage when viewing the cached document. Even todays date!

So anyone clicking on the link expecting my site is going to their site.

Another strange thing, and I am a novice here, is in the header of the 302 is a reference to a third site. This site is definitely doing the overture pfc scam. I have had several other sources check this for me.

Now, how this is related? I really don't know. That is what I am trying to find out.

I have contacted the offending site with no response as of yet. I have also contacted their service provider and registrar. No response as of yet. Good luck on this though, the site is located in .nl and the registrar is in .de.

Oh and of course I have contacted google, no response as of yet.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 6:13 pm on Sep 17, 2004 (gmt 0)

dirkz,

I finally got a respone from the offending site. I don't know, but maybe he is reading this thread. One of the reasons I will not go away.

Anyway...

He swears he is not using any blackhat methods on his site. He claims it is a bug in google. He claims he does no cloaking and that all references to my site have been completely removed from all aspects of his site and code.

Yet when I do a
cache:http://www.widget.com/link.php?id=5932
I get my homepage showing the documant as being retrieved on Sep 16, and the date on my page being Sep 15.

Now I am asking anybody, is this possible? Can this really be a bug in google? Is there any way to really check?

I don't want to start a campaign of reporting this site as blackhat unless I can be sure. It would be a bummer if what he says is true and I try to take him down. His site is listed in yahoo, dmoz, google and probably a lot more search engines besides.

And in the back of my mind is the fact that I get no responses from google on this. I need some advice on this one folks.

Thanks

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 7:25 pm on Sep 17, 2004 (gmt 0)

IF there are no longer any references to your site on his site, and IF Google is working correctly, then the next time google spiders his site, Google SHOULD update their cache.

IF that does NOT happen, it should't be too difficult to set up a few non-cloaked redirects and see how Google handles them.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 7:54 pm on Sep 17, 2004 (gmt 0)

IF there are no longer any references to your site on his site, and IF Google is working correctly, then the next time google spiders his site, Google SHOULD update their cache.
IF that does NOT happen, it should't be too difficult to set up a few non-cloaked redirects and see how Google handles them.

Okay, I'll buy that. I guess I don't understand why the dates are so recent. Isn't that an indication of either his site or my site being recently crawled? The cache date is yesterday.

iceman

10+ Year Member



 
Msg#: 25638 posted 10:05 pm on Sep 17, 2004 (gmt 0)

This thread is really informative. This same exact thing happened to a few of my websites about 6 weeks ago. I pulled my hair out trying to figure out what happened to my pages. After chalking it up to simple content theft, I dropped the issue.

Now, after reading everybody's posts, I'm going to reexamine a bunch of other sites that have seemed to drop recently.

Thanks everybody!

Patrick Taylor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 12:01 am on Sep 18, 2004 (gmt 0)

MAIA: Patrick, if you are still following this at all, did you check the cache on the pages you unintentionally hijacked?

Yes, I am following this thread - very interesting too. The results I referred to are from Yahoo mostly, and there is no cached page being offered in cases where my script replaces the page it's linking to.

Patrick

claus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 11:55 am on Sep 18, 2004 (gmt 0)

my3cents:
>> In message 2 you reply to Brett...
>> Obviously, at least one post was removed.

Yes, i remember that one now, nothing shady going on. As i recall it was just a misunderstanding, as mille wasn't quite clear on what the problem really was. After reading this and similar threads, it's pretty obvious what the problem was, but it wasn't really obvious back then in October.

Oh, and for readers of that thread, please think twice or more before firing off complaints like i recommended mille to do (this was one of those posts that i have regretted later) - at least try contacting the offending website politely first, that's usually better. In some cases this "thing" is not intentional at all, so the offending webmaster could be totally innocent (really!).


...a few thoughts...
I think this issue is all about SERP quality. To improve SERP quality, some measures has been taken to display non-blank pages (and/or snippets) in a number of cases where blank pages or url-only listings would be displayed previously (frames, redirects, meta, splash-pages, etc.) After all, the searcher will find that it's nice to see your wonderful widgets-page regardless if it's on your domain or not.

This has had some bad sideeffects, but until this mega-thread, the reports have been (relatively) few and relatively scattered. I do believe that the Google people can solve this issue, but i'm not sure if they want to, as there's always a trade off between "the right thing to do", and "the thing that gives the better serps" (here, "better" is not measured in webmaster metrics).

Anyway, it seems that abuse of some of these sideeffects is becoming widespread, so let's hope they decide to do something about it soon. Afaik, they have previously picked up problems from these boards and solved them, but of course this requires that it is really perceived as serious problems, not just the occasional bug. (so.... keep posting :-)

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 4:35 pm on Sep 18, 2004 (gmt 0)

This has had some bad sideeffects, but until this mega-thread, the reports have been (relatively) few and relatively scattered.
It's my belief that this is simply because only selected high traffic sites have been hijacked and that highjacker's have remained below the radar until now. It seems 301's, 302s, and meta-refreshes continue to work as they should for millions of sites.

I do believe that the Google people can solve this issue
I don't disagree. Google must conform to the HTTP Protocol and handle redirects as it specifies. But, assuming webdude's hijacker is cloaking, Google could simply ban the site based on their anti-cloaking policies. Will they? Time will tell.

kaled

WebmasterWorld Senior Member kaled us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 5:56 pm on Sep 18, 2004 (gmt 0)

Google must conform to the HTTP Protocol and handle redirects as it specifies.

Jesus once said "The Sabbath is made for man and not man for the Sabbath". If following the spirit or intention of HTTP opens a gateway to hell then it is probably not a good thing to do.

It is my experience that most problems have simple solutions, although they may sometimes be unpalatable. In this case, the simple solution is to index the pages that have the content. So if the content moves, a redirection must be placed on the old url to the new. This may not be entirely in keeping with HTTP, but it is clear, simple to understand and makes malicious exploitation of redirects impossible.

It does not require vast amounts of brainpower and hugely complex algos. This is a simple problem with a simple solution. All Google have to do is implement it. Their apparent inability to do so is pitiful.

Kaled.

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 8:48 pm on Sep 18, 2004 (gmt 0)

So anyone clicking on the link expecting my site is going to their site.

Another strange thing, and I am a novice here, is in the header of the 302 is a reference to a third site. This site is definitely doing the overture pfc scam. I have had several other sources check this for me.

Now, how this is related? I really don't know. That is what I am trying to find out.

Oh and of course I have contacted google, no response as of yet.

Again, IMHO this is not really a Google matter, and other than sticking with their guidelines to the letter, they're wise to stay out of the middle.

My opinion is that this hijacking business reflects the same kind of mentality as browser hijacking - which ALSO I believe is motivated by profit, namely PPC affiliate income.

I have contacted the offending site with no response as of yet. I have also contacted their service provider and registrar. No response as of yet. Good luck on this though, the site is located in .nl and the registrar is in .de

But Overture is in California. :)

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 8:53 pm on Sep 18, 2004 (gmt 0)

It's spreading

[webmasterworld.com...]

eduardomaio

10+ Year Member



 
Msg#: 25638 posted 9:03 pm on Sep 18, 2004 (gmt 0)

I've got the same problem on a specific page with a very specific topic... When I click on it I go to a strange website about Kazaa, Emule and other stuff like that... But my website is about cars...

worker

10+ Year Member



 
Msg#: 25638 posted 12:55 am on Sep 19, 2004 (gmt 0)

Any thoughts on this scenario:

A site uses a redirect URL to link to your site, and due to the issue we've all been discussing, the redirect URL is substituted for your sites main URL within Google.

When looking in Google for your site, you find the title snippet, and the description, and the redirect URL, instead of your own URL.

What I've described is basically what everyone else has spoken about. Here is what I'm curious about though.

The company that was using the redirect URL did not mean to have this happen, and they removed the redirect URL string, and replaced it with a direct link a few weeks ago.

Here is the question. What happens now?

My thinking is that Google will crawl the page that use to have the redirect URL link on it, and now they will find a straightforward link.

Will that mean that the redirect URL link, which is now no longer in place, except within the Google system, will dissappear?

If so, how long should it take before the redirect URL string is gone within Google, and the primary URL back in place, within Google?

If not, then what do you think will happen?

claus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:34 am on Sep 19, 2004 (gmt 0)

>> Will that mean that the redirect URL link, which is now no longer in place, except within the Google system, will dissappear?

Unfortunately, for Google it seems that URL equals Page

So, that redirect URL is now a non-existing "orphan page", although it has never really been a page, only a link. It will probably turn into a ghost URL (an url-only listing in SEPRS) and be buried in the "supplemental results" collection.

The only way to remove an url from Google is to use their Url removal feature [google.com] .. which, ironically, involves a page that is a dead link itself right now, but i have used it successfully in the past.

Your page will get its url back in ... well, it could actually be two months, but i'm not sure it will take that long.

Patrick Taylor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:54 am on Sep 19, 2004 (gmt 0)

What about this:
header("Location: $location",true,301);
within a php redirect script and where $location is the destination page? Currently the script is sometimes indexed in place of the destination page - it returned a 302 before I altered it.
worker

10+ Year Member



 
Msg#: 25638 posted 1:59 am on Sep 19, 2004 (gmt 0)

Thanks Claus!

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 5 [6] 7 8 9 10 11 12 13 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved