Forum Moderators: Robert Charlton & goodroi
The Problem
Hundreds of thousands of unique domain names having millions of subdomains and sub-dirs, running 2 distinct scripts (no idea which) on each domain. The one was outputting a page with only 5 sponsored links which is similar to the big Adsense rectangle and the other presenting a list of sites and the other outputting a page with several links to what appears spammy sites but each time you click the same link takes you to a different domain, including mine. All linking is done internally to a very long URL which is 302 redirecting to the victim sites.
The URLs of the sites targetting me:
302domain.com/keyword1.html
keyword2.302domain.com/
302domain.com/keyword3/keyword4-keyword5-keyword6.php
All the random domains I checked from a list I got using some DNS tools to reverse lookup the IPs hosting the spammer site are indexed, apart from some long subdirectories. Some of these sites have way over 15,000 pages in Google.
My hardest hit site had a TBPR of 4 (the same as the main sub-sections) and the homepage TBPR went down to 3, while the sub-categories remained the same. All rankings of the targetted sites were lost, while the only Google hits in my logs were from very obscure keywords only. Nearly the entire site into supplemental and things not looking good at all.
The Research I Did
This includes going through noumerous threads, some with hundreds of replies [google.com...]
However, after trying to dig a solution out of all this and failing miserably I decided to walk it alone.
The Possible Solution
I decided to test my main site and a couple of my other sites and try to move it away from the "penalty spot" by 301 redirecting my homepage to a homepage.htm page. So Googlebot was getting a 302 from the spammer site and then a 301 from my index page to the homepage.htm
This change was done back in Augoust and now the site seems to be working, as the site: was fixed and the main site finally got a TBPR of 5 for my main site and its homepage.htm after the recent Google update. For some keywords the sites rank high for the first time since November.
Also, for some reason, after doing the redirect to homepage.htm the spam domains started to *gradually* dissappear from my referrals logs.
On the sites that I did not do the redirect to the homepage.htm there are over 100 such referring sites which indicates that possibly it is not an algo update that did the trick, but the 301...
[b]Interesting Facts[b]
The spammer sites do not show up into my Analytics referrals. I can only find them out through Drupal's logs (http://drupal.org).
Matt Cutts has never answered such a question, even though I posted a few times on his blog asking for clarifications.
Google recently updated its TPR and possibly updated its algos, which means that I can't be 100% positive that what I did caused my site to appear or if it was the result of an algo update.
On a couple of my sites I have not done the 301 redirection and still no changes in Google's SERPs. It seems that the problems well exist (in my humble opinion).
Any ideas from other WW members would be appreciated, especially on what you have done to fight this spreading cancer. Is there something I am missing, or what helped revive your rankings?
Thanks in advance...
301 redirecting my homepage to a homepage.htm page.
An interesting defense. As a general rule redirecting the Home Page is not a good thing -- I've seen site vanish from the SERPs when a search is done for their domain name. However, in your situation I can see that it might be a help.
Please do keep us posted on this, and how it holds up over time.
[edited by: CainIV at 7:10 pm (utc) on Oct. 10, 2006]
Interested to hear what counter-measures others have taken to fight this problem. I am certain I am not the only one hurt by this.
They have the company name as the link text and it does a 302 redirect to my / page.
[edited by: mcskoufis at 7:26 pm (utc) on Oct. 10, 2006]
Is it possible that your page just simply outranks the spammers copy because of your pr going up?
Or maybe Google got a clue and treats an unguarded 302 as a link from page1 to page2 for ranking purposes. Who knows.
One question of many I have is did you by some happenstance actually correct a canonical issue on your site by doing the 301?
What I just noticed from my Sitemaps account is that one of our most competitive keywords that we just reappeared in the GRP (google result pages) much stronger this time. However it is not the homepage that ranks it is one of the subsections. It was no6 back in November 2005 and now it is no1, above some very authoritative sites.
How are you detecting that Google got there via a 302?
With that said though, remember the description of this thread... It could well be a google algo update.
I am sure that googlebot is visiting... All the homepages of the domains and subdomains are in the google index with 0 GTPR... Some have over 190,000 such pages indexed by google.
I am dead certain that googlebot is finding them and crawling frequently, because it is only the Google rankings that have been hurt.
I just want to make certain that you didn't also correct another canonical issue.
Alas, there is more than just the www/non-www problem.
Alas, there is more than just the www/non-www problem.
Even though I am not posting here frequently, most of my time daily is spent on reading the ever-endless threads of Webmaster World and have fixed any canonical irregularities. And this for the past 5 months or so. I keep constantly refreshing the Google search news section here throughout the day for hints.
Believe me I've done extensive research on the issue, because I originally thought that the 302 could not in any way hurt me.
I have made sure that each page is only represented by a single URL only and other things which have not yielded any results about my site. It is funny cause we have links (not paid) from some top websites on the Internet with GTPR of 9 and 10. Don't want to say which because it will probably make it easy to understand what my site is about and possibly which site is it about.
How are you only redirecting the Googlebot on the 302 only requests?
I am redirecting everything to the homepage.htm (visitors + bots). Did this as a desperate measure in August and with the recent update the homepage.htm got a PR of 5 at last with some hits from Google to deep pages of my site.
I just want to make certain that you didn't also correct another canonical issue.
Exactly (assuming theBear meant "create" another canonical issue)! By sending a 301 from the domain root to another url on the site, you can create a serious canonical confusion. One result can be that your domain is not returned when people enter it into the Google search bar -- and lots of people do use Google as if it were their browser's address bar.
For this reason, a 302 is normally the better choice in my opinion, when redirecting the domain Home Page to another url on the same domain. It also might serve your purpose here -- it is still a redirect chain.
Sheesh, I can't believe after all these years that Google still appears to have trouble with the cross-domain 302 issue. It must not be as simple as it seems to me: 302 to a url on a different domain should mean index the TARGET url, right? Only when the target url is on the same domain should the source url of the 302 be indexed.
[webmasterworld.com...]
After the latest google update I am the top result for "my company name" although 2 days ago it fluctuated to the 4th result and today it is back no 1 again.
I am worried cause now things are looking better... no1 for "my company name" and supplementals start after the 10th page of results for site: operator, instead from the first.
These guys target the top results for certain keywords including the authority sites. From reverse-iping the IPs of some random ones (they are over 100,000 domains/subdomains, I reckon (with no evidence yet) that they somehow trick google into sending traffic to a third domain which is a big corporation where they have one page for each domain. This site is separated into categories (hundreds of them) and they target relevant sites with the ultimate goal of driving traffic to those pages.
Unfortunately for us, even though we should have, our pagerank is very low (as opposed to the authority sites which they target) and thus makes our site really vulnerable. The authority sites don't seem to be missing anything... They still rank high for our main keywords.
All happens from a single IP address which has all those sites hostges including the big corporate one.
It is a real pitty cause a site run by very few people (less than 5) which has managed to gain the trust of so many visitors and members and is regularly linked in blogs/forums/authority sites because of it's quality of content is hurt so much. We are fine competing against the very big boys out there and 5 months onwards I can not find a solution to the google problem. It is so unfair.
When presented with a link to URL A, where URL A issues a 302 redirect to URL B, Google can still index the content at URL B and show it in the SERPs as belonging to URL A.
I found just such an example only yesterday.
That proxy was not issuing 302s in its normal operation.
I spent some time reading the java source code for it.
His solution to the problem involved 302ing to an error page which would eventualy free the pages that were copied by the proxy.
One thing we should not forget is the other search engines like yahoo and msn and redirects. How will it effect their indexing?
If only Google could take some appropriate action and provide some insight for the webmasters who are genuinely hurt by this problem.
Not sure what you mean "by proxy", but since you read this thread I take it that it is what happening with my case.
It is worth noting that each time MC was available for webmaster questions or when GG instructed to send an email to support with "canonical" as the subject I did send in questions. MC spoke about everything else apart from the spam issues, which at the end of the day are under his "authority" in Google. From support I got a reply saying that "There is almost nothing a competitor can do...".
The continuing silence of Google reps regarding this issue, smells trouble for me and I seriously think that they have yet to find a solution for this (personal opinion here and with no offence to the Google folks).
Surely there must be someone who has the same problems. I've had a good friend who has managed to rank sites in the top10 of their chosen keywords and keep them there look at my site, but after spending a lot of time he is suggesting to me to get a different domain, cause he can't figure out and can't explain this behaviour (as far as google is concerned).
His solution to the problem involved 302ing to an error page which would eventualy free the pages that were copied by the proxy.
Interesting suggestion. However, programmatically this is one of the issues puzzling me most, as I have yet to find a way to block only hits originating from those ips. I get patterns of domains sending me traffic from the same ip address. They have over 100,000 hosted on that IP address.
My knowledge of regular expressions is non-existing, thus making it impossible to develop some clever .htaccess code which can do what theBear is suggesting.
I'm just tossing out some questions and so far haven't read anything in the _mail_ here that indicates you are being held hostage by a proxy (one should look up exactly what a proxy is and should note that a real proxy identifies itself as such). There are many real proxies on the web and there are _OTHER_ things that _RESEMBLE_ proxies. There are also proxies that are incorrectly setup that can cause a bit of trouble.
Just so you know, I wouldn't expect to see a 302 at all from a proxy.
One thing we should not forget is the other search engines like yahoo and msn and redirects. How will it effect their indexing?
My yahoo and msn rankings have not been affected. Part of the "my company" name is a very generic keyword with very stiff competition. In both MSN and Yahoo we are in the top 10 for this generic keyword. I am using only ethical white hat SEO techniques and have sites ranking in the top 10 for generic keywords for the past 3-4 years. Those sites simply can not be hurt as they have become authoritative in their field.
Trinorthlighting, what I am trying to say is that I am using techniques which I've learned over the years (reading this and related "authority" sites). And my strongest criticism of Google is that "hey, you are forcing me into black-hat seo".
(unless this tool is so inaccurate... they list the first 2000 though)
I am a bit worried about making such changes now that there seems to be an ongoing google update with different site: and link: results
Then by all means, wait and be sure that you still need a fix before implementing anything new in your case.
Some of what I post is meant to clarify the issues involved for others who my read this thread now or in the future. Redirecting the domain root with a 301 is a pretty drastic thing and can have unplanned for results over time -- that's what I wanted to warn about.
I wouldn't expect to see a 302 at all from a proxy
Agreed.
It's really important not to be trying to cure your "case of the bird flu" without first verifying that you actually HAVE the bird fllu.
------------
IF (a big if) someone knows for sure that their results have been hijacked by another domain pointing at them with a 302 redirect, there is one drastic but effective counter I read about here a few months ago. Here's how I remember the post:
That person put up a robots.txt on their own domain that blocked all spidering. Quickly they went to the Google removal tool and entered the hijacker's domain -- but since Google was following the 302 redirect, they grabbed the robots.txt file from the target (hijacked) domain instead of the source (hijacker) domain. This effectively removed the hijacker domain from Google for 6 months. Now one more VERY quick step. Watching for googlebot's request, this person then quickly removed the new robots.txt as soon as it had been served to googlebot.
Definitely a desparate move and not for the faint of heart. If it goes wrong, your own domain can be irrevocably dropped from Google for 6 months. And if slurp or another spider hits withinthe critical period, you may be blocked on another search engine as well.
But before that I want to raise one issue towards the Google folks. I am a mid-age recently MSc awarded computer scientist (it was awarded to me 10 days ago after submitting my thesis)... I operate several sites and the one worstly hit was the one I had been working on for 14 hours a day for over a year with the only day off being Christmas eve...
My girlfriend left me months ago, I see my very close mates once every 2 months for a couple of hours, spend most of my time here on this site going through miles of discussions and interesting points which I need to be focused on in order to understand and even my family here in Greece I hadn't had the chance to fly over from UK to see them the past 6 months.
Ok I am not complaining about my busy occupancies. But who will compensate my 6 months or so of research into finding out how a spammer can fool Google? And what about the every penny I've put into my brightest project to go wasted?
Noone.
What would my sites look like if I didn't have to spend so much time on a single issue? Wouldn't that benefit your users Google?
My point is that there are problems in computing.. You can not build the perfect and failproof system, even if you had rocket scientists working for you.
A bit of help and a bit of "de-classifying" information when there are so many genuine websites severely damaged and ground-breaking ideas from small well educated "enterpreneurs" going wasted in the sandbox leaving only the big boys unvulnerable is a disgrace.
And by disgrace, I mean that you Google had maintained a profile which we all liked (and we supported you and still do so) and we started thinking that we would never see again. Not because this is the case, but because YOU let us think so by your continuing silence on the issue.
If only you realised how some people survive in this world... How about those who can't even get a hint of what a website is really like? Those who start something creative and a stupid issue like this turns them away...
And at the end of the day this has been going on for years. You could have got in contact with those posessing the knowhow of how to trace things like this. You could have spent more on improving your search and less on other things. You could have not stamped out creativity and good will.
That's all about mcskoufis' rant...
Good night to all and many thanks for the insight you've put in. A recap of things to check is required here. And yes, an answer from Google about whether my method is having any effects or it was an algo update that did the job would be great. Or what we need to do to get this cancer away...
Ok I would go with hijack rule number 1.
Are you serving your domain on an ip address as well as the site url?
ie. can your domain be reached via an ip address and the ip address shows the content of your homepage. Your homepage should only be reached from your full url. If by typing an ip address you can also reach homepage you have a vulnerability.
[edited by: Pirates at 3:09 am (utc) on Oct. 11, 2006]
