homepage Welcome to WebmasterWorld Guest from 54.196.63.93
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google News Archive
Forum Library, Charter, Moderator: open

Google News Archive Forum

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 [5] 6 7 8 9 10 11 12 13 > >     
Dupe content checker - 302's - Page Jacking - Meta Refreshes
You make the call.
Marcello

10+ Year Member



 
Msg#: 25638 posted 11:35 am on Sep 7, 2004 (gmt 0)

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

 

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 11:12 am on Sep 15, 2004 (gmt 0)

There has to be a reason and a reward for doing something like this kind of hijacking, same as there is with browser hijacking. It's deliberate, and it has to be more than just a technical exercise. Here's some food for thought

[webmasterworld.com...]

Patrick Taylor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 11:47 am on Sep 15, 2004 (gmt 0)

I read it, thanks. The whys and wherefors of the thread are beyond me technically, and I'm not in those competitive areas, otherwise I would devote more time to the issue. I can see how passing an outgoing link through a meta-refresh page can wreak havoc to the site being linked to but don't really understand the destructive process, deliberate or otherwise. Having seen in my tracking system that some of the pages on sites I've built for other people are being replaced in search engine results by the php redirect script on my site I'm obviously concerned if I'm inadvertently doing the same kind of damage to others - hence I took an interest in this thread. Redirects are common enough, I know - I wanted to know if I was doing it correctly. But never mind. It's been an interesting read and food for thought.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 12:35 pm on Sep 15, 2004 (gmt 0)

I don't know if this has anything to do with the problem, or if it is the reason any g reps have been silent, but google itself uses redirects. If you do a search on the regular google site, the result are as follows...

[google.com...]

Run that through a most header checkers you get a
200 - OK - Successful

request that goes to the correct site.

The header checker I use actually follows the path and records ALL header info along the way. When I run it through this one I get...

#1 Server Response: [google.com...]
HTTP Status Code: HTTP/1.0 302 Found
Cache-Control: private
Location: [widget.com...]
Set-Cookie: PREF=ID=5bcbbd8133e3c10a:TM=1095251202:LM=1095251202:S=YCP_LsM5k7Uv7lZP; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com
Content-Type: text/html
Server: GWS/2.1
Content-Length: 152
Date: Wed, 15 Sep 2004 12:26:42 GMT
Connection: Keep-Alive
Redirect Target: [widget.com...]

#2 Server Response: [widget.com...]
HTTP Status Code: HTTP/1.1 200 OK
Date: Wed, 15 Sep 2004 12:26:03 GMT
Server: Apache-AdvancedExtranetServer
Set-Cookie: PHPSESSID=071c5a12a2be547c54596f47aefe7219; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=a8aac90657fd9451e9c89af0b38ac783; path=/
P3P: CP="CURa PSAa PSDa CONi OUR DEL BUS ONL STA DSP IDC COR"
Connection: close
Content-Type: text/html

It could be a little more dicier to fix then an algo change. I am sure the the G is aware of the problem and looking at a fix. You might have to wait a crawl or two before you see results though.

I still have had no response from any emails to google.

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 12:38 pm on Sep 15, 2004 (gmt 0)

>>So what does it take to fix this?

IMHO what it would take to fix what I believe is the root of the whole problem has nothing to do with Google. What might help is for the PPC companies paying out affiliate income to those acquiring clicks through fraudulent means to be held accountable for not establishing guidelines for their affiliates.

How it is now, the PPC company gets income from the clicks they get, their affiliates get income from the clicks they get by hijacking pages, and none of those parties seems to care as long as they're getting their revenue. It's a win situation for them, they're getting money however it's acquired.

The only losers are the innocent webmasters whose traffic is getting stolen by fraudulent means. It isn't a Google issue any more than browser hijacking for PPC income is a Microsoft issue. It's a PPC affiliate issue, and the rock bottom issue is that the ultimate responsibility lies with the integrity and accountability of the PPC companies themselves.

It may be a dead issue if it's offshore entities being dealt with, but if it can be traced to affiliate ID numbers and it's a company in the U.S., that company needs to be informed in detail. They can't be ignorant enough not to know what's going on, but it can't hurt to make sure they do know - with specifics.

Marcello

10+ Year Member



 
Msg#: 25638 posted 1:04 pm on Sep 15, 2004 (gmt 0)

After reading more on the Internet and talking with some expert friends overhere, I have to agree with Brett that:
"there is so much more going on here that meets the eye"

And am now thinking in the direction that a simple Meta-Redirection on another site can not replace an Index-Page and steal its position.

I presume now that the other page(site) is also using some server-site "Black Hat" and "cloaking codes" together with the Meta-Redirect.

I would be thinking in the direction that the site is presenting one version of their page to the Search-Bots, making the Bots caching the Re-directed page instead of the Re-Directing page.

And presenting another version to users/browsers which simply includes the Meta-Redirect ... of course I could be completely wrong.

Also by reading on the Internet I found out that this is not a Google problem alone, it's a general problem with all Search-Engines currently.

I am sure that Google and all the other engines are aware of the problem and are working very hard towards a solution, but Rome was not build in 1 day.

Interesting reading materials here:
301-and-302-redirects [rankforsales.com]
Meta Refresh And Search Engines [netmechanic.com]

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:15 pm on Sep 15, 2004 (gmt 0)

Marcia,

I read the link you posted at

[webmasterworld.com...]

I sure am glad that I do not pay for clicks.

The only losers are the innocent webmasters whose traffic is getting stolen by fraudulent means.

On the directories I have found, I am actually thinking of contacting as many sites as I can to inform them of how their pay for clicks are being abused. Might be a daunting task though. There seems to be literally thousands of sites. What a scam these directories have going.

To everyone,

No more stickies please. I am only going to answer mods and seniors. I am NOT going to give specific URLs so don't bother asking. The offending sites are easy enough to find if you know what you are looking for.

If you are a site that is paying for clicks, I would study your logs closely. And any other data that is at your disposal. If you are getting lots of referrals from directory type sites, I would definitely check those sites out. Unfortuantely, I do not know how logs would show a clickthrough using a redirect or meta refresh, but be aware. It could be you are paying for nothing.

jbauder

10+ Year Member



 
Msg#: 25638 posted 5:37 pm on Sep 15, 2004 (gmt 0)

this is interesting though I cant say I fully understand it all yet ... I will bet the months I spent trying to figure out which Google filter my sites tripped is nothing more than getting pages jacked

outland88

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 8:13 pm on Sep 15, 2004 (gmt 0)

The only thing I can say for sure is about every time I come across some traffic-hijacking scheme it more often than not relates to Adsense. The latest Iím looking at is a well-known site that purports to help kids with their homework. The site has hijacked about 1/3 to 1/2 of every indexed page on the Internet and in DMOZ. By copying this much content from other legitimate sites they appear high in Goggle on millions of keyword searches to sell Adsense. Plus by the sheer size of the site it becomes an authority site. At first glance it appears legit but as you look more deeply the breadth of what they call content extractions (content theft is what it really is) is astounding. Most of these sites would die a quick natural death but Adsense and content theft breathes new life into them.

Maia

10+ Year Member



 
Msg#: 25638 posted 8:38 pm on Sep 15, 2004 (gmt 0)

<<After reading more on the Internet and talking with some expert friends overhere, I have to agree with Brett that:
"there is so much more going on here that meets the eye"
And am now thinking in the direction that a simple Meta-Redirection on another site can not replace an Index-Page and steal its position.

I presume now that the other page(site) is also using some server-site "Black Hat" and "cloaking codes" together with the Meta-Redirect.>>

Maybe in some cases. There are webmasters in participating in these threads who have admitted to inintentionally hijacking other site's index pages.

See Rick M's posts:
[webmasterworld.com...]

kaled

WebmasterWorld Senior Member kaled us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 25638 posted 11:18 pm on Sep 15, 2004 (gmt 0)

there is so much more going on here that meets the eye

The only reasonable translation of this into plain English is "The problem is even bigger than any of you think".

My guess is some alien piconites have escaped from Area 51 and, intent on world domination, they are invading cyberspace through fibreoptic cables - and of course they are intent on taking control of Google first.

So it's either an invasion by alien piconites or Google techs have stuffed up. No doubt Google would have us believe that the piconite theory is the more likely of the two.

Kaled.

BigJay

10+ Year Member



 
Msg#: 25638 posted 1:38 am on Sep 16, 2004 (gmt 0)

Patrick Taylor: Try This.

<?php
header("HTTP/1.0 200 OK");
header("Location: $URL");
?>

Patrick Taylor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 2:41 am on Sep 16, 2004 (gmt 0)

Try this

Thanks for that, but it didn't work either. $location is what identifies the various URLs, so I need that in "header()" I think.

Anyway I probably missed the point of this thread as it turned out, since it appears to be less about a simple meta-refresh problem than a PPC ripoff. Because I see my redirect script appearing in search engine results, at first I saw it as maybe one and the same issue (WRONG).

quotations

10+ Year Member



 
Msg#: 25638 posted 4:16 am on Sep 16, 2004 (gmt 0)

if this is the technique for hijacking, what is the actual trigger that causes google to think the hijacker's page is the real one? Is it the fact that the hijacker's page is identical?

Several of the sites doing this hijacking are combining it with cloaking.

Each cached page of their site is a copy of a page on my site.

When you search for items which are on my site, the hijacker shows up in the SERPS. If you go to the cache, it is a copy of my site. If you go to the actual url, it is a site which has a similar topic to my site but no content. It requires you to buy a membership to see the contents of their site, but if you do that, it is actually a hard core porno site.

I pointed this out to webmaster@google.com using GoogleGuy in the subject and the reply stated that I could "go pound sand."

That is not a direct quote but almost.

I then asked them this:

Are you saying that cloaking is now an acceptable practice?

Their response basically said that they have no problem with what this site is doing.

charlier

10+ Year Member



 
Msg#: 25638 posted 5:39 am on Sep 16, 2004 (gmt 0)

Hi Patrick,
You are right just change the $URL to $location in the code I posted. The $ thing is just a variable that holds the URL that you are redirecting to, in your case its called 'location'. I just posted it with the variable name I use in my scripts. Sorry for the confusion.

Macro

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 11:57 am on Sep 16, 2004 (gmt 0)

After reading more on the Internet and talking with some expert friends overhere, I have to agree with Brett that:
"there is so much more going on here that meets the eye"

I don't know whether that does or does not exclude the original post itself? ;)

kaled, it could also mean that there are some hidden - and less obvious - agendas behind some of the posts. I wish I was clever enough to know which ones and figure out what those agendas are.

quotations, I feel for you but I thought Google was serious about DMCA issues. Do they not see one here? Also, it is not the case that all cloaking is penalised, AFAIK. Maybe, they're skirting a fine line. It is well known that Google like to do things via algo and automatic systems rather than hand-editing. If that is the case it may well be that they are taking this situation into account while tweaking their systems to catch such culprits automatically.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 12:49 pm on Sep 16, 2004 (gmt 0)

Okay, it seems that deleting the link on the offending page does not work. At least in my case. If someone could explain to me why or how this doesn't work, I would appreciate it.

Here is what I got...

The link now ranks #5 for a good phrase. The link is...
[widget.com...]

This link belongs to a directory that allows you to login and edit your listing. When I found this link, I immediately logged in and deleted it from the offending directory (3 weeks ago). My thinking was that I would sit back and wait for the link to disappear and see if my site came back.

Today, not only did it not disappear, but it actually gained a few spots in the SERPs! When I run...
cache:http://www.widget.com/link.php?id=5932 it shows MY homepage. So what is the deal with that? And the date on the cached file is yesterday and the date on the cached page is the day before! (I date all my pages)

So now I am totally ticked. The original meta refresh is gone. It has now been replaced by a 302 redirect. It ranks where my site should be. The 302 goes to their home page but the cache shows my page. It seems that deleting my link from thier directory did not work.

So what the heck do I do? I suppose I could add my link back into their directory, that way I could at least get some google referrals again (all referrals from the G have now disappeared - gone - nada - zilch).

And to make matters even more irritating, I found another site doing the EXACT SAME THING! I also deleted that link which led to a meta refresh, which is now a 302, which now shows my home page as being cached, yet goes to their home page.

Meanwhile, I have 2 subpages that rank in the 500s. That's it. All other ranking for my site is gone except for these redirects. Two months ago. My site was ranking anywhere from 50 to 10 depending on the phrase. That's all gone but these other links still remain.

Still no response from google.

dirkz

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:53 pm on Sep 16, 2004 (gmt 0)

If you want google to react to this problem, I guess you'll have to hijack google.com and redirect it to a p*rn site.

Would be interesting to see if in that they case they would still favor algorithmic handling (which won't help for a long time) or nuke the site manually

worker

10+ Year Member



 
Msg#: 25638 posted 4:09 pm on Sep 16, 2004 (gmt 0)

This issue is now making it out into the mainstream. SearchDay just included a link to an article on Pandia about spammers manipulating Google and stealing websites. The article ends by stating that Google has not addressed the problem.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 4:56 pm on Sep 16, 2004 (gmt 0)

Could you please sticky me the URL?

Thanks

worker

10+ Year Member



 
Msg#: 25638 posted 5:07 pm on Sep 16, 2004 (gmt 0)

Just go to Pandia and look at the articles for today on the right hand side of the page.

webdude

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 8:49 pm on Sep 16, 2004 (gmt 0)

Okay, something wierd is happening.

The SERPS haven't changed, but I got hit by googlebot about 100 times today. Not a crawl, just the robots and home page.

Maybe this is the beginning of the end. I'll keep you informed.

By the way, no googlebot in weeks until today... we'll see.

Maia

10+ Year Member



 
Msg#: 25638 posted 9:09 pm on Sep 16, 2004 (gmt 0)

Webdude,

Did you try the googlebot@google.com email address, yet? I read the article from Pandia, and I really don't think Google will tell anybody anything via email after that. They have to be aware of that article. If you search "Spammers hijack web site listings" the article is all over the place, now.

Your situation is almost identical to mine. Except that I couldn't manually remove my link, because it kept giving me an error-so I emailed the webmaster of that site to remove the link and they did. They also made a snide comment about banning my IP from submitting to their directory so I wouldn't have this problem again. I know it isn't the same directory, because they use a different linking structure than the example you posted.

After they removed the link, which actually meant the link now goes to a page on their site which then goes to a 404 error page...they did move up in the SERPS for the search terms I found them in.

Eventually, it did get dropped from those search results, altogether. It took more that a few days, though.

I didn't lose any other pages in Google, just the index page they had redirected to, which at the time was showing my page in the cache. My index page was completely out of Google for a time, but I can find it in there now. Also, during this whole thing Googlebot was only hitting my index page and not crawling further, but today I noticed they did crawl some other pages.

As I stated before, after all this, with the link removed from their directory--if you search on just my domain name, that link they were using now shows up instead of my domain name. Which it didn't do before!

Like you, I am wondering if this is because the link still goes to their site before going to the 404 error page.

claus

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 9:30 pm on Sep 16, 2004 (gmt 0)

Been busy for a while, nice thread... So, we're back here again, huh?

>> If this problem's been around so long it's a shame someone didn't press Google on it pre-IPO (Macro #78)

Not exactly pressure, but at least mentioning the problem: What about those redirects, copies and mirrors? [webmasterworld.com] (May 3, 2004)

>> And after searching I found many threads, all with different variations of the same problem. (Webdude #7)

In the thread above, msg #8, you'll find a nice collection of 24 threads dating back to... june 16, 2003

Maia

10+ Year Member



 
Msg#: 25638 posted 10:53 pm on Sep 16, 2004 (gmt 0)

Well, that answers the "I wonder if Googleguy knows about this" questions from other threads:

[webmasterworld.com...]

Robert Charlton

WebmasterWorld Administrator robert_charlton us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 25638 posted 1:06 am on Sep 17, 2004 (gmt 0)

In the thread Maia cites just above, claus says...

It's simply a Google bug, albeit a very annoying one.

This resonates, because when I spoke with a Google search engineer about the meta refresh redirect problem at SES San Jose in August, 2003, he used the exact same words. "Hmmm, it sounds like a Google bug," he said. I know it's a big system, but that was over a year ago.

my3cents

10+ Year Member



 
Msg#: 25638 posted 1:37 am on Sep 17, 2004 (gmt 0)

claus,

I found something in your collections of threads that I find interesting. In a thread titled:

"Not sure but I think it is Page Jacking" October 9th 2003 ( [webmasterworld.com...] )

In message 1 mille reports a page jacking

In message 2 you reply to Brett...

Obviously, at least one post was removed. You must have made a point to cause Brett to remove his own post!

... traveshamockery I say!

canucks1980

10+ Year Member



 
Msg#: 25638 posted 1:48 am on Sep 17, 2004 (gmt 0)

I think its pathetic how everyone screams foul when something doesn't go their way. As soon as someone is positioned ahead of them the word "lawsuit" billows like an atomic cloud of chaos. Lets face the facts, google, yahoo, msn, etc are not PERFECT. They will not deliver PERFECT results 100% of the time. Yes they blunder and yes sometimes its not fair, but suing your competition isn't as fun as BEATING them. Instead of legal action build a site similiar to theirs, optimize it, legalize it, and screw them at their own game. Sometimes results speak louder then words. Nothing is permanent so don't assume anything. Try to be nice to each other, there is enough money out there to make everyone happy. What this entire post comes down to is greed. Is this an example you want to set for your children?

Patrick Taylor

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 25638 posted 1:56 am on Sep 17, 2004 (gmt 0)

... screw them at their own game...what this entire post comes down to is greed

I beg to differ. A few members here had their pages screwed up (in at least one case, a hobby site) apparently by other sites exploiting what possibly turns out to be a "Google bug", and they wanted to find out why. Screwing people at their own game would be, in this case, like becoming a pickpocket just because one had one's own pocket picked.

I hope the members affected get their sites back to where they should be when the game is played fairly. In my case I'm pleased to say my php redirects are now returning a 301 (thanks to charlier and BigJay).

Maia

10+ Year Member



 
Msg#: 25638 posted 3:08 am on Sep 17, 2004 (gmt 0)

What this entire post comes down to is greed

I see "Search Engine Optimizer" is what you listed as your occupation in your profile. Bye!

quotations

10+ Year Member



 
Msg#: 25638 posted 3:12 am on Sep 17, 2004 (gmt 0)

quotations, I feel for you but I thought Google was serious about DMCA issues.

Apparently not serious enough.

They told me that they do not care what the content of the pages they index is and that they would do nothing about it.

Marcello

10+ Year Member



 
Msg#: 25638 posted 4:10 am on Sep 17, 2004 (gmt 0)

While I was sleeping......

Woke up this morning to find an answer from Google to my DCMA complaint about the duplicate copy of my Index-Page at foo.com.

The answer comes down to:
Google has removed "the page in question" from the Google-Cache and the archived information for this page ... but searches for widget will still return this page's title and URL until our bot has revisit this page again.

On the demand why "My Index-Page" was banned and replaced by "My Index-Page under foo's URL", the answer is that individual responses are not given, followed by the canned text that eventual penalizations can have many causes, followed by the list of "What Not To Do" as stated in the Google guidelines.
(for which I dont worry as my site is clean)

Furthermore, due to my DCMA complaint to other Mayor Search-Engines, the complete site of "foo.com" is since this morning (my local time) gone from their SERPS and "My Index-Page" under "My URL" has reappeared in its former glory-position.

Things are starting to move .....

This 389 message thread spans 13 pages: < < 389 ( 1 2 3 4 [5] 6 7 8 9 10 11 12 13 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google News Archive
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved