Forum Moderators: Robert Charlton & goodroi
It is a local info site (not yellow pages). The url structure was key-phrase.example.com.
After doing some resarch it appears that I received a minus 30 penalty. Reading up on the penalty it seems that this penalty can be caused by:
1. Dodgy links
2. Over optimization
3. Thin affiliate sites
4. Duplicate content
The site has no dodgy links. The links to the site are all unsolicited. It is a borderline authority site with Wikepedia links, many BOTW links (included by editors, not me) and other spontaneous links from a variety of sources.
So I don't think its a link issue.
I took a hard look at whether or not it was over optimized and made some changes to be on the safe side.
I had 3 instances of keywords in every title and changed it to just one.
Same with meta-description.
I left the <h1>key phrase</h1> at the top of each page and key phrases where appropriate on page.
There is no hidden text, hidden links or any other such nonsense.
There were two affiliate ads on each page. One with javascript, the other html. I removed the html affiliate ad - again, just to be safe.
Finally, I had recently added archive pages with older info that was paginated. The paginated pages all had the same title and description. I had thought Google was smart enough to see that each page had different content but evidently not.
I disallowed all of the archive pages via robots.txt, added no-index, no-follow to each page.
I had been planning to change the URL structure to http://www.example.com/key/phrase/ later this month so that I could use Webmaster Tools. I had also been considering moving it to another, more appropriate domain.
So, rather than waiting to see if the penalty goes away (the site is a large percentage of my income), I moved it to the new domain with the new navigation structure. The new domain already had PR3 but wasn't doing much for me.
The new domain is being crawled VERY heavily by Google right now.
Questions....
1. Is there anything else I can or should do?
2. Assuming I have corrected the issues that caused the minus 30, will the penalty go away and, if so, how long will it take?
It might help the discussion if you describe how you decided you had a minus-30 penalty, when in fact your not fitting much of the profile.
You've probably already done this, as you've obviously done some research, but, if not, take a look in the Hot Topics [webmasterworld.com], which is always pinned to the top of this forum's index page. In the Penalties section, this thread might be helpful as a guide for checking whether you are penalized...
Am I penalized? What type of penalty do I have? [webmasterworld.com]
From what you describe, I think the archive pages might have been a factor.
I disallowed all of the archive pages via robots.txt, added no-index, no-follow to each page.
Note that robots.txt negates the noindex,nofollow robots meta tag, which you also added. The robots.txt prevents Googlebot from spidering the page, so it won't see the tags. I'd recommend one approach or the other.
You may well have hit Google with too many changes all at once right now. I don't think I would have moved to a new domain until I'd seen what else worked. Many of these things take time to kick in. What's the timeline in the process you describe... how long has all this taken?
Is there anything else I can or should do?
I don't think I'd make further changes on the site right now, but I would make sure that your old domain is properly 301 redirected to the new one. I usually redirect the pages that had external inbound links (check on both Yahoo Site Explorer and on Google) and leave the rest for Google to sort out. With this many changes, it may take awhile.
Others may have more elaborate instructions for how to handle your 301s.
If you know what other sites were linking to the old domain and can contact them, I'd contact them and ask them to change their links to the new domain or specific deep page they'd been linking to before.
To be honest, I'd be slightly concerned that you may have gone further than necessary with the changes you've already made (mainly with the domain move). If the link profile is as solid as you say, then it would be somewhat unusual for the site to suffer a penalty, except in the face of more serious transgressions.
In some ways, switching domains might even create the wrong impression. My preferred approach would be to have made much more gradual changes, in order to isolate an area that may have been a problem, so: that the same thing would not occur again.
Prior to any change to navigation or to on-page text, I would try to rule out any technical problem that may have triggered any issue (starting with things like the URL footprint, status codes and the like).
I'd also be watchful that the problems were not caused by external sites (e.g. via duplication, or as a result of problems with the link profile), in which case they could quite easily affect the site again, and are more likely to affect a site that has moved domain.
I fully appreciate that there can be a business case for dramatic change - and dramatic change can achieve dramatic results. But it also carries a much greater degree of risk: it could exacerbate overlooked or misunderstood problems.
It might help the discussion if you describe how you decided you had a minus-30 penalty, when in fact your not fitting much of the profile.
It was pretty easy to identify. Because the site was set up as http://locale.example.com - with 40K plus locales - we had a lot of locales that enjoyed good rankings for their respective keywords.
Then, all of the sudden, Google traffic nearly stopped. There are zero referrals coming in from the top spots. The only referrals that come in are from 30 or higher.
Also, the site's home page which is on www.example.com fell from its #1 position for the site name.
The drop was sudden and complete - with every local site suffering - which tells me that the problem is with example.com and not any individual page(s) or subdomain(s).
I would make sure that your old domain is properly 301 redirected to the new one.
Note that robots.txt negates the noindex,nofollow robots meta tag, which you also added. The robots.txt prevents Googlebot from spidering the page, so it won't see the tags. I'd recommend one approach or the other.
Is it problematic to have it in both places?
I would try to rule out any technical problem that may have triggered any issue
I'd also be watchful that the problems were not caused by external sites (e.g. via duplication, or as a result of problems with the link profile), in which case they could quite easily affect the site again, and are more likely to affect a site that has moved domain.
Initially I was concerned that it might be proxy hijacking or scraping but searching for unique phrases doesn't bring up any likely suspects.
Thinking more about this today, I think it may have been a combination of the subdomain navigation structure with the added problem of the archive pages with duplicate titles and descriptions. Perhaps one or the other would have been OK, but maybe the combination of the two was a tipping point and made the site seem spammy to Google.
On the bright side, there have been a few of the pages indexed from the new domain as of a few hours ago. While the site doesn't appear for any of the popular keywords yet, it does appear for [locale] [unique to this site] searches in the #1 position which leads me to believe the penalty hasn't (yet?) been transferred to the new domain.
[edited by: Hcb1 at 8:04 pm (utc) on July 8, 2008]
[edited by: Robert_Charlton at 8:20 pm (utc) on July 8, 2008]
[edit reason] delinked subdomain example [/edit]
I put the meta tag in the page itself because I wanted to make sure that Google didn't index the page if it came from a 301'd page off the first domain. I put it in the robots.txt also so that Google wouldn't even bother crawling the pages from the links it found on the new domain.Is it problematic to have it in both places?
With the robots meta in place but no robots.txt, Google will not include references to the page in its index.
With both in place, Google won't see the robots meta on the individual pages, so if a page somewhere else on your site or on the web links to one of your pages, the link might show up in the index... on a site: search, perhaps.
The contents of the page should be blocked from indexing in either case.
If there's a page you absolutely don't want referenced, though, I'd recommend usually the robots meta tag only, not blocked by your robots.txt. I do this for things like ad campaign landing pages, where I'd prefer that competitors not have easy access. With regard to your situation, it may not make any difference.
Robots.txt disallow - page not spidered at all, but the URL for the page can appear as a URL-only entry in the SERPs.
Robots disallow meta tag - page is spidered but nothing appears in the SERPs. URL does accumulate PR and can pass it on to other pages that it links to.
Yahoo will "invent" a title for the robots.txt disallowed entry using the anchor text from one of your incoming links, as long as that anchor text is NOT some sort of generic "click here" or similar.
.
*** with 40K plus locales ***
I would be concerned at any site with 40 000 sub-domains. Often fits the profile of a prolific spammer.
.
*** I put the meta tag in the page itself because I wanted to make sure that Google didn't index the page if it came from a 301'd page off the first domain. ***
The crawler does not live-follow redirects.
The bot has a list of URLs to access. It fetches the content and status of each one, and that data gets stored in a database. In the case of a redirect, the data about where the redirect points would be stored, and Google would direct-access the new URL on the next run of the bot, not follow it live at the time of the original spidering. The spidering would follow the rules found in the copy of the robots.txt file that another part of Google's system would have downloaded in recent days.
Questions....1. Is there anything else I can or should do?
2. Assuming I have corrected the issues that caused the minus 30, will the penalty go away and, if so, how long will it take?
Coming back to your original question [2.], that's a big question and one that deserves an answer. I don't know and i wish i did.
I guess we're talking two types of full SERP's return from the penalties:
1. reconsideration requests and
2. algorithmic release
Personally, I have little faith in the reconsideration request, because:
- you rarely receive a response from Google, leaving you even more anxious and uncertain.
- it leaves you open and exposed to the discretion of a Google human editor that may use "interpretation" of Google's guidelines in an inconsistant way - and that's dangerous. There are always variances in human interpretation.
- you may have missed something and just strengthen your problems
Therefore, i think the focus must be on algorithmic restoration.
I have heard that these penalty situations can get "stuck" and the situation lingers, sometimes for months and years. Something needs to provoke an "untripping " of the filters. Just "tidying" things up may be a bit too passive for anything to occur.
You've got to remember, Google often tolerates a lot of wrong things before it looses patience and penalises a site. Therefore, in reverse, it's possible that the lazy G might not be fast to move things in reverse and get you back in the SERP's .
So what are the best ways ?
Who has had success and can testify to it ?
I for one would very much like to know what the variables of success are here.
Problems that are purely on your site are more often released by algorithm, in my experience. Cloaking may be one on-site exception. Clean that up 100% and ask for reconsideration. But off-site issues? If you've been trying to manipulate backlinks, and the debris of that is still all over the web, then you've got a much more challenging job, and the algo alone will probably not release you from the penalty.
If you been creating spam blogs, get them offline. Feeder networks and link farms? Pull out of them if they're not yours, or take them offline if they are. Wild card subdomain spam? Just drop it, all of it. Redirects of any and all devious purpose should vanish.
If the penalty is already on you, then don't even think about trying to hold on to those parts that "Google couldn't possibly have detected."
If you've been up to something even deeper than the above, then just burn the domain and start over. You've played in the realms of disposable domains, and now it's time to dispose of it.
A page on one of my sites ranked #2 for years for its keyword phrase (behind the Wikipedia entry on the phrase.)
In May, my page disappeared from the index, replaced by an exact dupe of my page on a French server. (The dupe used my logo, AdSense code, header, footer -- everything.)
I DMCA'd Google and it removed the offending page in June. A week later, my page returned to the SERPs, but indexed under a parked domain on the same server, a URL I haven't used for that page since 1997.
Then, last week, that URL disappeared from the server, and now there's *another* page from a different website, with my text copied word-for-word, in the top 10.
I'm thinking about DMCA'ing that page, but wondering if Google will penalize me for filing multiple DMCAs. I have to say, though, if my page's copy is so good that multiple sites are copying it and ending up on the first page of the Google SERPs, why can't Google just allow the original back into the index?
FWIW, the page has been online since 1996, and at the same URL since 1997. It has more than 6,000 unique backlinks using the keyword phrase, all organic. (The site's won a slew of awards over the years.) No paid links. Only ads ever placed on the site were AdSense.
I filed a reinclusion request via Webmaster Tools, but have heard nothing.
Thoughts?
Some types of penalties will not be released in a purely algorithmic fashion - especially those that were placed manually from the start
How can you tell a manual from algorithmic penalty ?
If your team, or your contractors, are not coming clean about their actions, then that's an issue to resolve internally. But if you are confident you know what kind of so-called-SEO actions have been taken in your business, then you also know whether you've crossed the big black line into manual penalty land.
When in doubt, submit the reconsideration request anyway, even if you suspect it's a algorithmic effect. If you're already penalized, then paranoia about comminicating with Google is a self-defeating action. They're not even as much of an adversary to you as the tax man is. They want to rank good sites - that's their business!
I guess the assumption is that the smaller the "minus", the less severe are the issues behind it.
You're looking for a simple rule that does not exist, Whitey. The penalty we've been calling -950 is algorithmic. The penalty we originally called -30 was (at that time) manual. Every drop in rankings by xx positions is not a "-xx penalty", it's often just a convention of speech among webmasters.
If the penalty means you rank at -30 for a search on your own domain name, then that's a bad one
Do you mean "mysite.com" or just "mysite"
However, being at #31 for a search on your exact domain name, including the TLD (extension) is a big trouble sign and almost always indicates a penalty - and at least in the past, a manual one.
