To stop the 301 or not? - penalty recovery strategy and help

[dipper]

I originally posted this on the Google Webmaster Forum .. got 0 responses.

Hi,

I manage a NSFW website selling toys+lingerie (on DVD, movie, or anything like that - yes there is a huge distinction) -

Anyway, we were hit with a manual link penalty a few years back, we resolved that (confirmed by G), but since then we've never really recovered. We have a disavow file with about 1000 domains in it, which we add to at least every month, trying to shield ourselves from potential problems.

In July 2014 we changed URL's to the current URL from the old NSFW URL with a view to leaving the past behind, copying the disavow file over to the new domain. For the first month we bounced back, and thought we might be okay, but alas since then we've been on a steady slide back down, and way below where we were in July 2014 even.

In the last 9 months we've taken massive steps in removing old products (10000 down to 2500, with 301 redirects on old to related new), consolidating products into groupings (small, medium, large, black, white, red, green etc), adding plenty of movies and videos to the site to showcase products, encouraging customers to come back and post reviews on the site, reducing our server load times, implementing good cross linking between products to ensure customers can better find related products. We've also run in-depth scans on the quality of our copy/content, using copyscape, screamingfrog and siteliner, and replacing, removing, adding to any and all descriptions which looked bad/thin/ugly or simply didn't sell the product well. We've learnt our lesson, we've really improved beyond site of where we were. Time on site has increased, pages viewed, and we think customer engagement has improved. The site has been mobile friendly for over 12 months.

We've also been doing a lot of PR, and interviews and talking about the right things, and tapping into our network of customers to gain seriously good quality links (very few, but good quality) from top notch magazines and publications. We have all of the new good quality links point to our new domain, we have no bad links pointing to this domain directly, and it seems only bad links pointing to our old domain (although they are disavowed anyway).

My only thought is that the old domain is a ball and chain holding us back, and possibly dragging over a undeserved penalty.

So, this all sounds great, BUT we've seen zero improvement in rankings or traffic. In fact since December our traffic has actually declined.

My questions are ->

1. should we cut the old sites 301 redirect to the new site? Is the old domain a chain and ball around our neck?

2. Also - If we made the old domain parked/null, would the old-url's hurt us with duplication? What about other people who scraped our old site, and we switched over the new site later than they scraped?

I'm interested in the thoughts of the knowledgable guru's here!

[goodroi]

Roughly how many links are pointed to the new site?
Other than Google what is generating traffic for your site?

[dipper]

@goodroi

re: links - The new site has about 60 good quality links.

re: traffic - In Australia Google dominate in terms of the search engine of choice - 97% of our organic traffic is from Google, even though our rankings are fair at best when compared to Yahoo and Bing where we are ranked very well. In terms of overall traffic, 37.5% Google organic, 25% email marketing/newsletters, 20% Google Adwords, 5% referrals from shopping comparison sites + those big mags I mentioned, 10% direct + a few % social. Would not expect social to be high given our market tbh.

[dipper]

I should also note .. I'd say there are virtually no good links left pointing to the old domain - its purely historic. I wonder on the 2 questions though what we should do .. and I should clarify with #2, we would have a single page on the old-site, which had "We've Moved + a link to the new site" on it.

[Shai]

Here are a few questions/pointers that we always go through... If I remember any more, ill repost.

1. You say you created a disavow file with 1000 of domains. How did you create a master list of the links pointing to your site? I.e what sources did you use? You should have used at least, Majestic, Ahrefs, OSE and Google SC (previously knows as WMT... I hate Google for doing that!)
2. What is the % of disavowed domains compared to the master list of urls?
3. Have you mainly removed domains or urls?
4. Have you submitted the disavow file to all 4 versions of the two domains? (www and non-www)
5. How brutal/honest were you when creating the disavow? Were you tempted to leave any links that would be counted as dodgy in any way?
6. Have you done a page -> page redirect?
7. Using Copyscape is good but not always up to date or perfect. You should also always check for problems by copying random sentences into a Google search box and wrapping it with a double quotes. Is your site number one for the chunk of text? are there lots of copies of it around? Is there a message from Google at the bottom of the page saying it has removed some results from the serps as they are found to be duplicated?

Hope this helps, Shai

[dipper]

@Shai

1. yes, 1000 domains - 50% would be nofollow domains, 50% dofollow domains. Created by compiling a master list of all links from Majestic, ahrefs, GWT and Moz OSE. Run through linkreasearchtools to spit out a bad list.
2. 95%+ .. any good domains and good links we have moved over to point directly to the new URL. Only bad links point to the old domain. but they are disavowed.
3. domains mainly
4. yes http+https, old+new
5. we removed ourselves from judging them using linkresearchtools as our analysis tool. disavowing anything (if you know the tool) which was not green.
6. yes -> page to page redirect. But with the removal of 7500 products in our clean up, this meant 301 pages to new/altered pages.
7. yes, of course we do that, but with 2500 product pages its impossible to always keep track. We do DMCA requests when we find them using copyscape, or manual checking.

Still you didn't answer my original 2 questions though .. to stop the 301 or not?

[Shai]

Stopping the 301 is completely dependent on the steps you have taken before hand and how confident you are that the site is clean of any underlying problems.

Two things that ring alarm bells.

1. How quickly did you do the redirect after you submitted the disavow file? We normally recommended that you wait at least 5 - 8 weeks to give a chance for the pages the links are on to be re-crawled so that Google can then discount them from the link graph. From your description, it sounds like this was not done and hence the penalty flowed through to the new domain.
2. When checking links, there is no substitute to checking them manually. Yes, its a lot more work but the only way you can be 100% sure enough links have been removed. We ALWAY find some really bad links that automated tools have missed out.

With regards to the redirect, we always find that things just come together much quicker if the redirect exists. (obviously, after making sure all of the above is taken care of in a proper manner) Not really sure why. May be the age of the domain or some sort of other authoritative signal that you just don't get with a new domain.

If all of the old domain pages have now been de-indexed (and they should have by now) then there is no problem with duplication if you decide to remove the redirect and the old site completely.

[netmeg]

I'd probably cut the 301s myself, but I'm like that.

re: links - The new site has about 60 good quality links.

How were these acquired? Did they come about naturally?

[goodroi]

You removed 1000 links and replaced them with 60 links and expect the same ranking results?

I don't know your specific details and I am not saying that your disavow or the new links are good or a bad idea. Personally I would not want 1000 garbage links and have seen 60 good quality links power a site to the top. I have also seen some people take shortcuts which results in great rankings till they get penalized. They remove the poison and then wonder why the traffic hasn't returned. The traffic hasn't returned for those people because they didn't replace the poison with enough healthy ranking signals.

This might not be a 301 issue, this just might be that you haven't developed enough good ranking signals.

[dipper]

@Shai
1. it was about 2 weeks at most before we switched over to the new domain. Note: we were not in penalty at that time, not what Google call a penalty anyway.
2. yes I agree, accept if you've disavowed 99% of the links to get out of a manual penalty. The remaining 1% was healthy and good, and moved over to the new domain since then. We actively monitor links, so any missing would have been picked up and checked since then.

@netmeg
We did a lot of PR and press, hit up some contacts who were in the media/press and just started talking more about our industry, where it is moving, etc etc. Basically we took a far more personal approach to gaining links, with respect to wanting to change our market, become a market leader and some quality links flowed. Yes, naturally.

@goodroi
In short, yes I would, absolutely. The quality of those 60 links is far, far superior to the 500 dofollow links I described above. Most of those 500 have been disavowed now anyway, and the remaining good links (1%), moved over to point directly to our new domain. The volume and quality of those 60 exceed the quality of the inbound links for the top 10 sites currently ranking for our big terms - being objective about it.
I don't think we have taken any shortcuts, we've been really proactive and inter-personal in talking about our brand, industry and products, and in this way gaining some links just sort of happened. Most were from talking on the phone, and through interviews, stats, etc etc. We might be in a titillating industry, but sometimes this sparks more interest in understanding purchasing habits, or the types of products available.
It might very well be that we still haven't got the good ranking signals needed for the size of site we are. Or maybe removing the 7500 products meant our quality rating declined ..

The question is still .. to stop the old domain 301'ing to the new domain? and will scrapers get the edge, given our content would have moved domains?

[Shai]

@Shai
1. it was about 2 weeks at most before we switched over to the new domain. Note: we were not in penalty at that time, not what Google call a penalty anyway.
2. yes I agree, accept if you've disavowed 99% of the links to get out of a manual penalty. The remaining 1% was healthy and good, and moved over to the new domain since then. We actively monitor links, so any missing would have been picked up and checked since then.

You may not have been under a manual penalty, but the temporary jump in rankings after the redirect is a clear sign that there was an algorithmic penalty/filter (call it whatever you want). The subsequent drop indicates that the penalty has gradually flowed through the redirect. May well be because you did not give the disavow enough time, or maybe because the disavow was incomplete or even because the links were not the problem in the first place.

It is well documented, reported and observed that penalties are layered and you can well be under layers of penalty. Peel one off and you have another one underneath it to get rid of.

This is a really important point for anyone who may be reading this thread in search of assistance with recovery. The identification of what it is that hit your site is paramount. If you have established that the penalty/filter is link based then the next step is to do the link clean-up in a methodical (an in my opinion, manual) manner.

The biggest tip I can give anyone at this stage is to wait. We normally wait for anything between 4 and 8 weeks. This is extremely important. You do not redirect to the new domain until you are confident that the majority of links have been re-crawled by Google. Depending on the amount of links you have, you can use various tools to crawl the urls and check to see the % of pages that contain the links that have been re-crawled by Google since the submission of the disavow file. We normally feel its safe to do the redirect when we hit around 75% re-crawl rate but that can vary depending on the severity and quantity of links. If you do the redirect too early, your site will see a recovery, drop and you will find yourself in a situation where you have used up your shot of recovery with that domain. i.e if you don't have a domain you can use for another go, you are now stuck until the refresh comes along.

[dipper]

@Shai - I probably didn't make that clear enough, the manual penalty was lifted in July 2013, and we moved the site to a new domain in July 2014.

[Shai]

That would indicate that either you did not completely get rid of all of the links or that there is another problem with the site. If you are happy that everything I said has been covered without any shortcuts then I would look towards content related issues. (in which case, there is no reason to get rid of the redirect)

[dipper]

@ Shai - yes I would tend to agree. Thanks for your thoughts!

[tangor]

Considering all the effort expended on the new site, I (ahem, probably not in the majority) see no reason to keep the 301. You've moved the links you want. Gaining (I hope) new links as well... there is no reason to keep the old to, as you say, be a ball and chain.

At this point you have no clear area of action by attempting to keep the bad in play to go with the new. And you are building for Google instead of the user. Indications that Bing and Y are doing well.... so concentrate on the USER and the site.

[dipper]

@ tangor .. I had exactly the same thoughts - opinion is divided though, like most things seo related. Why even consider keeping the old pointing to the new, if there is a chance it could be hurting?! .. if there is nothing of value on the old, then move on. I know from reading in the past JohnMu said keep the 301 for a minimum of 180 days - its nearly 360. What worried me the most is scrapers publishing my content more recently than the new domain. I just have to keep on top of my DMCA requests - its a game of whack-a-mole though.

[netmeg]

As I said, I'm on the side of disconnecting the 301s myself.

[McMohan]

Considering all the effort expended on the new site, I (ahem, probably not in the majority) see no reason to keep the 301

I will go with tangor. If you have removed majority of the links that helped the old domain rank in the first place, no reason why should you retain the 301.

As far as duplicate content is concerned, you could consider simply robots.txt disallowing the old domain and remove the site using URL removal tool. Do a random sampling search of say 10% of your pages with "some random sentence" in Google and see if your pages come #1 consistently. If they do, you need not worry about duplicate content issue.

Did you check your analytics to see if there are a large number of pages with very poor usage metrics? High bounce rates, small time on pages etc?

[dipper]

@McMohan.

update - I've stopped the 301. Good idea on the robots, I've added that in. I was simply redirecting all sub-pages to the homepage, and that page was a simple, "We've moved to <HERE>" type page.

As part of the removal process I mentioned above it was to remove long gone OOS products, and try to remove a lot of dead-weight on the site. As mentioned we started to include video's, adjusted our page layouts, grouped pages together into configurable items (red, green, blue, etc) .. upshot is its hard to know given no recent Google Penguin or Panda updates as to whether this made a difference. I do know that our traffic has declined since Jan though.

We recently implemented the adjusted bounce-rate into our GA, so I cannot compare < March 2015, but here are some numbers.
For all sessions in the last 30 days (total > 30,000)
Bounce rate avg: 12.21%
Exit Rate: 16%
Time on page Avg: 1.00 minutes
Avg session duration: 4.44 minutes
Pages/Session: 6.23

All sessions include email newsletters, Adwords, etc, but organic is about the same rates (slightly lower on pages per session at 5.18, and 13.81% bounce rate) but on a smaller sample, 35% of total.

Our homepage has a bounce rate of <11%, with exit rate of < 30%, time on page 1.18 minutes.

While this is not delving down to page level as that would be hard to give here, if we had a significant problem on many pages (as we now have 2500-2600 products) I'm sure the numbers above would show, yeah? .. previously we had a lot of pages with much higher (when weighted) bounce rates, and much higher exit rates.

Anyway, you tell me .. any alarming numbers?

[McMohan]

Anyway, you tell me .. any alarming numbers?

None that I can see. The metrics from your numbers look great to me. Not many sites come with such numbers.
As someone mentioned earlier, it might be due to lack of links pointing to your new domain, albeit of good quality. It all depends on the level of competition and what Google deems right to rank, information vs commercial. If you don't see other sites in your niche with similar backlink portfolio ranking in the first page, the niche might demand better link portfolio to rank higher. In the health segment for instance, you will see a particular set of sites such as webmd ranking consistently across a wide range of keywords, and you know you don't have much hope with a smaller site with not so great a backlink profile.

But the long and short of it is, you might just as well remove the 301 without any direct or collateral damage. I too am in a similar situation trying to rank a site now for over a year, but the returns when it finally does is great, so you keep working towards making the website better and demand attention from Google.