Dupe content checker - 302's - Page Jacking - Meta Refreshes

You make the call.

[Marcello]

My site, lets call it: www.widget.com, has been in Google for over 5-years, steadily growing year by year to about 85,000 pages including forums and articles achieved, with a PageRank of 6 and 8287 backlinks in Google, No spam, No funny stuff, No special SEO techniques nothing.

Normally the site grows at a tempo of 200 to 500 pages a month indexed by Google and others ... but since about 1-week I noticed that my site was loosing about
5,000 to 10,000 pages a week in the Google Index.

At first I simply presumed that this was the unpredictable Google flux, until yesterday, the main index-page from www.widget.com disappeared completely our of the Google index.

The index-page was always in the top-3 position for our main topics, aka keywords.

I tried all the techniques to find my index page, such as: allinurl:, site:, direct link etc ... etc, but the index page has simply vanished from the Google index

As a last resource I took a special chunk of text, which can only belong to my index-page: "company name own name town postcode" (which is a sentence of 9
words), from my index page and searched for this in Google.

My index page did not show up, but instead 2 other pages from other sites showed up as having the this information on their page.

Lets call them:
www.foo1.net and www.foo2.net

Wanting to know what my "company text" was doing on those pages I clicked on:
www.foo1.com/mykeyword/www-widget-com.html
(with mykeyword being my site's main topic)

The page could not load and the message:
"The page cannot be displayed"
was displayed in my browser window

Still wanting to know what was going on, I clicked " Cached" on the Google serps ... AND YES ... there was my index-page as fresh as it could be, updated only yesterday by Google himself (I have a daily date on the page).

Thinking that foo was using a 301 or 302 redirect, I used the "Check Headers Tool" from
webmasterworld only to get a code 200 for my index-page on this other site.

So, foo is using a Meta-redirect ... very fast I made a little robot in perl using LWP and adding a little code that would recognized any kind of redirect.

Fetched the page, but again got a code 200 with no redirects at all.

Thinking the site of foo was up again I tried again to load the page and foo's page with IE, netscape and Opera but always got:
"The page cannot be displayed"

Tried it a couple of times with the same result: LWP can fetch the page but browsers can not load any of the pages from foo's site.

Wanting to know more I typed in Google:
"site:www.foo1.com"
to get a huge load of pages listed, all constructed in the same way, such as:
www.foo1.com/some-important-keyword/www-some-good-site-com.html

Also I found some more of my own best ranking pages in this list and after checking the Google index all of those pages from my site has disappeared from the Google index.

None of all the pages found using "site:www.foo1.com" can be loaded with a browser but they can all be fetched with LWP and all of those pages are cached in their original form in the Google-Cache under the Cache-Link of foo

I have send an email to Google about this and am still waiting for a responds.

[dazzlindonna]

Bloggers have recently had a lot of power in gaining the attention of the masses. Perhaps a concerted effort of blogging about the problem is due. I have blogged about it in the past, and I blogged about it again today. Those who have blogs, consider doing the same.

[crobb305]

I have a friend in the media who says this would certainly be an interesting news topic. I hate to suggest a press release because Google may be working on this problem. The issue is that they are remaining silent and that they have had this problem since 2003.

And legitimate businesses are suffering at the hands of malicious hijackers.

[siteseo]

As I posted in another thread, I believe part of the duty of the Supplemental Index is to weed out dupe pages/hijackers. The definition G gives for the SI is:
"Supplemental results are triggered on a relatively small number of queries for which Google's main index does not provide many results. Because this index is still in testing..."

So clearly it is still being tested.

I can only say that many pages that duplicate my content (especially affiliate links) have only recently been moved to the SI.

[crobb305]

The fact that the site:mysite.com command is still showing spammy domains being associated with my site tells me that nothing is changed. Yes, they are in the supplemental basket, but they are still incorrectly being shown as part of my site.

[walkman]

"The fact that the site:mysite.com command is still showing spammy domains being associated with my site tells me that nothing is changed. Yes, they are in the supplemental basket, but they are still incorrectly being shown as part of my site."

agree with you. It doesn't matter where they are. They were on the supplementals before I deleted them but I was still penalized. The supp. links should've been ingored, instead I got penalized too.

I still am penalized but hopefully Google will update itself. I deleted some links a week ago and the last one 3 days ago. The people who inadvertently linked with Meta or 302 have been very cooperative and nice.

[energylevel]

I still think we need some clarity on this issue as the exact nature of these redirects and how to diagnose if they are hurting us, maybe some of us are assuming these redirects to be the problem when the penalty is for something entirely different (I'm just playing devil's advocate here you understand).

I've heard to diagnose do a search for:

inurl:www.yourdomain.com
allinurl:www.yourdomain.com
site:www.yourdomain.com

.... now I've seen a site where the allinurl search is showing a few redirects but the site: search is clean so are we saying if the redirect doesn't appear in an site:www.mydomain.com then it's NOT an issue?

I've seen a few redirects similar to this now in allinurl searches:

www.externaldomain.com/somescript/php?url=www.mysite.com

I've been copying and pasting the URLs into the 'View HTTP Request and Response Header' tool at web-sniffer.net and seen a variety of things going on with 302's, meta refreshes, javascript redirects.

The latest one I've seen shows a HTTP Status Code: HTTP/1.1 200 OK with my domain as the host and the content as my homepage content. Does this mean Google might think it's a mirror of my homepage and dish out a duplicate content penalty?

I'm getting to the stage now where I've no idea which redirects may be hurting or not.....

If anyone has a proven list of actual methods that are known to be hurting the sites they redirect to could they please post here or start a new thread. If they think the info is too sensitive please send me a stickymail.....

Has anyone managed found a method of getting these redirects removed from Google's Index quickly when they can't get cooperation from the offending site?

[walkman]

"Has anyone managed found a method of getting these redirects removed from Google's Index quickly when they can't get cooperation from the offending site? "

I would try to get email the offending site's host and if that fails, send a DCMA complaint to google. Your content is indexed under their domain. It's just as simple as that. Good luck

[eddy22]

My site has approx 500 pages but site:mysite.com shows over 1000 pages on google.
Any advice/tips will be appreciated.

thanks in advance,
eddy

[rocco]

to find dup content also use:

allintitle:
search for unique text appearing on your sites with quotes: "some aldja aldjfjlkj a whatever"

[hunkydory]

I have found a page on a site that uses this to "link" to my site: "<script>document.write('<me'+'ta·http-equi'+'v="refresh"·content="0;'+'url=http://www.'+'my-site-in-link.'+'com/">');</script>"

I tried emailing them to ask them to remove it but the email just gets bounced back and the whois info is no good either.

I have also done a google inurl search as mentioned in a post above and that returns loads of results for sites that are nothing to do with me (4 times as many results as there are pages on my site).

What can I do now- my site has gone from top 5 on several keywords in different search engines to not being listed anywhere in google, MSN or Yahoo