| 4:45 am on Aug 18, 2008 (gmt 0)|
If confusing chains of redirects have occurred in the past, how can Google be best informed to cancel or remove any records that they may inadvertantly hold, where they may have been previously confused ?
Could redirect chains be a potential cause of a penalty filter being applied ?
My thinking is that any redirects we do in the future will be allowed to settle in for say a month and then replaced with 404's once Google demonstrates it has reindexed the new URL's . Would this be a better procedure , or would another en masse redirect potentially destabalise a site?
| 4:54 am on Aug 18, 2008 (gmt 0)|
|pageoneresults - Any chance you could share the specific nature of your learning mishap for those of us still under eternal apprenticeship. |
Sure. Way back in the very beginning when we were first getting into ISAPI_Rewrite on Windows, we made some mistakes in the rules for our initial test site. Those mistakes caused a major loop within a relative URI structure where everything got appended to each other. The bot basically came into the site, hit the first destination and due to a screwup in the rewrite rules, would sit there and just append, append and append. We caught it immediately but it was too late for the indexing. Took a month to get that cleaned up and we never looked back.
I'm also learning that no matter how much you test, something comes up after the fact. They are usually minor challenges but they are there and not found during the initial testing. Working with redirects is an ongoing process and one that requires quite a bit of finesse and knowledge! Being on Windows makes it that much more challenging. :)
| 5:02 am on Aug 18, 2008 (gmt 0)|
|If confusing chains of redirects have occurred in the past, how can Google be best informed to cancel or remove any records that they may inadvertantly hold, where they may have been previously confused? |
I'd look at using the URI removal tool. I'd also look at using Webmaster Tools and Sitemaps to assist in the process. You have to provide instruction and Google gives you an extra option in the process, Sitemaps. Use them to your advantage.
|Could redirect chains be a potential cause of a penalty filter being applied? |
Redirect chains? Anytime more than one redirect comes into the picture, you've added "another" element for the bot to figure out. I can't say for certain, but I've seen multiple redirects lead to challenges in some instances. I mean, if I visit a destination and there are 4 or 5 server headers being generated, something may not be right. I'd have to start asking myself, how many of these redirect thingies can a bot handle before it says enough already! :)
|My thinking is that any redirects we do in the future will be allowed to settle in for say a month and then replaced with 404's once Google demonstrates it has reindexed the new URL's. |
Wishful thinking. :( Once you do a 301, it is forever.
We're using 410 Gone now with new implementations. At some point, we're hoping the crawlers start obeying the 410 status and stop treating 410 as 404!
|Would this be a better procedure, or would another en masse redirect potentially destabalise a site? |
No more redirects except those that are required to fix what is broken. Too many changes like this in short periods of time can have a negative impact "I think". You need to be real frugal in the use of redirects in the overall scheme of things. Too much of something is usually not a good thing, especially when discussing redirects.
| 5:16 am on Aug 18, 2008 (gmt 0)|
So how do folks with products that constantly change name, re appear or disappear handle it on an ongoing basis?
aka Shopping Sites , Travel Sites , Event Sites etc etc - It's very normal for the same widget to be rebranded as a different name.
| 5:27 am on Aug 18, 2008 (gmt 0)|
|So how do folks with products that constantly change name, re appear or disappear handle it on an ongoing basis? |
I'm sure many will respond and provide their advice. It is all going to be relative to quite a few things about the product. If something is going to reappear after it disappears then a 302 might be suggested.
|aka Shopping Sites, Travel Sites, Event Sites etc etc - It's very normal for the same widget to be rebranded as a different name. |
I've seen things handled a variety of ways. Custom 404s, 302s, 301s, helper pages, etc. I've also seen setups that "learn" and evolve with the user. ;)
| 5:56 am on Aug 18, 2008 (gmt 0)|
|Once you do a 301, it is forever |
..... sorry my post may not have been clear.
.... but you can replace a 301 with a 404 once the new URL has indexed Y / N ? Would this be one way to fix the confusion?
| 6:02 am on Aug 18, 2008 (gmt 0)|
|but you can replace a 301 with a 404 once the new URL has indexed Y / N ? |
Me? No you cannot replace with a 301 with a 404. Unless that is what you want. If there are external references out there, and there are, the 301 needs to be forever. Until all, and I do mean all, of the external references have been updated, the bots will continue to request the original. I know, it sucks, but hey, gotta do what is best. In this instance, if you 301 something, you cannot remove that 301 for quite some time. I say forever at this point. Give me a couple more years and ask me then. ;)
| 6:33 am on Aug 18, 2008 (gmt 0)|
Page is correct, the 301 needs to be there as long a external references still point to it - aka links.
If you are a trooper and can physically get every inbound link changed to the new url, then it would be safe after a few months to remove the 301 altogether. Replacing the 301 with a 404 only serves to cut off any link value the original page had (and likely any rankings)
| 6:40 am on Aug 18, 2008 (gmt 0)|
|Replacing the 301 with a 404 only serves to cut off any link value the original page had |
Yes ... this is exactly what i mean. The benefit of doing this seems to mean that Google no longer has to try and follow the old redirect chain once it knows which path to follow, which would otherwise be the cause of it's confusion and overall site indexing issues , such as tanking in the SERP's with penalty filters.
So I'm saying "cut it altogether" and introduce new URL's that index cleanly. Better to loose the link value and any IBL benefit's to those old redirected pages if it is going to allow Google to fully understand the correct URL's.
It seems to me from my experience and other posts that Google cannot reliably handle extensive chains of redirects , even if WMT Sitemaps is updated.
Y / N ?
[edited by: Whitey at 6:44 am (utc) on Aug. 18, 2008]
| 6:50 am on Aug 18, 2008 (gmt 0)|
Why do you think there is a penalty? That word is used to loosely around this neck of the woods. ;)
With all this redirecting you have going on, I'd be more apt to look at the technical foundation to make sure things are in order. Again, too much of this stuff over short periods of time may not be a good thing.
This whole "redirect chain" causes the hair on me neck to stand up, at attention! Think of them as "layers" in the foundation. I've seen some overkill in layering of redirects. "I" wouldn't feel safe in that type of environment but I'm conservative when it comes to this whole redirect thing. My thinking is that you should only need to deal with redirects when something is moved, in limbo, or gone. And then of course there are redirects that occur based on UA/IP e.g. sessions, etc.
| 7:13 am on Aug 18, 2008 (gmt 0)|
|Why do you think there is a penalty? |
Because the only alterations on a site I'm observing have been with redirects, consolidating content from several sites onto one , then further adjusting the content to others.
No keywords , even unique ones are above page 4.
The structure of the site under observation is the same as others that rank well. The content of the site is also of the same value, albeit different from the other sites.
No questionable backlinks to this site.
This is why i say it has a penalty filter applied to it. And the only observable other difference is that it is a .com , unlike the others with country specific TLD's. [ Non of the redirects involved country specific TLD's.]
| 1:43 pm on Aug 18, 2008 (gmt 0)|
Whitey I have a question on the 301's you consider caused a filter to get tripped.
Were the 301's done by mass folder, htaccess rewrites, or IIS from one individual page to the other?
| 2:45 am on Aug 19, 2008 (gmt 0)|
The chains are fairly lengthly. Probably 7 big sites , consolidated onto one
About 24 - 12 months ago htaccess rewrites were used.
Then we changed platform over to .NET and used individual pages to perform the redirects. I don't recall how many redirects were involved [ although we have records somewhere ] , but we're talking movements in the order of 15 - 30,000 on an originally 150,000 page / URL site.
All the new sites are considerably smaller - max 45,000.
Would the URL removal tool in WMT help remove the confusion if we applied it to all the old sites ?
| 4:26 am on Aug 19, 2008 (gmt 0)|
|consolidating content from several sites onto one |
I would be looking very strongly at this element of the puzzle.
I there was a redirect chain in place, and there are links point to the original source, I would amalgamate the chain to redirect from the oldest link (first piece of content on the chain) to the new source of the content. This means that the original link to the original document is preserved.
At the same time, trying to manually move the better links by manually contacting those sources and asking them to update their links to the new location can help expedite the stabilize the ranking process.
[edited by: CainIV at 4:31 am (utc) on Aug. 19, 2008]
| 1:03 pm on Aug 19, 2008 (gmt 0)|
CainIV I don't believe consolidating the sites is the issue. I suspect from Whitney's read the first redirect was ok most likely they used the htaccess rewrites to 301 the pages was fine and most likely ok. I believe the change in code from ? to .Net is the cause of his problems. I am not sure how they did a manual 301 with that many URL's and being DB driven I am sure this was possible.
Whitney what platform were you using before the .Net switch and was the site moved to another server at this time?
| 8:38 pm on Aug 19, 2008 (gmt 0)|
Prior to .Net we used Apache Server , and yes it was a different server.
| 8:50 pm on Aug 19, 2008 (gmt 0)|
Whitey, normally people are making the switch the other way, from .NET to .php or some other technology.
I'll assume that you've got a quality .NET programmer on board? Why? Because .NET out of the box FAILs in many areas when it comes to SEO. While an excellent programming language, it FAILs miserably in how it does things and the amount of code it generates to do those things.
Did you ever do a page to page comparison? I mean, did you ever deconstruct a previous .php page, and then deconstruct the new .net equivalent? What did you find? Probably a Text to HTML Ratio that is way out of sync with the .php version?
How many redirects? Did I see mention of 300,000 somewhere? Expect anywhere from 4-6 months for initial recovery based on my experiences. And, that is all going to be relative to a "bunch" of other factors.
What's your VIEWSTATE look like? Did you take stuff out of Postbacks so that is was indexible? Did you leave stuff in Postbacks so it isn't indexible? All sorts of questions.
| 9:03 pm on Aug 19, 2008 (gmt 0)|
Whitey I think we have now found the real cause of the filter. I feel it wasn't the 301's that caused the issue but a combination moving the site to a new server, different code, and on top of that redoing the 301's caused the problems your now facing.
Then we have the question was the site moved to the new server correctly, were all the folders kept and url's remained the same.
I would now begin to see if I could get some help as pageoneresults suggested as there may be things going on now with the new set up causing you problems.
| 3:45 am on Aug 20, 2008 (gmt 0)|
I'm sure the good folks on these threads that dig detail would appreciate me me tidying up my comments on this thread, as will newbies who are trying to follow the giste of this 301 issue and related factors.
There is also a tie in to the management of duplicate content [ and the related threads ], because inevitably tidying up sites with dupe problems often involves 301's.
See Hot Topics / Duplicate Content - [webmasterworld.com...] ]
So I need to clarify this post better with an approx calendar as indeed I may have confused some of you [ my apologies ]. However, I do think it will be helpful for folks to follow , because this experience can help share some of the issues created through extensive redirects, something i haven't seen too much commentary on . So here goes again :
Probably I have confused you with my statement “Probably 7 big sites , consolidated onto one” . To make things easy my focus is only on the core .COM site . The other sites are relevant , but they were involved in a lot less redirect activity into them or between them - though some did occur.
Firstly there were separate sites [ approx 2004 ] , then a consolidation [ late 2006 ] into .COM , then a progressive split into separate regional TLD sites [ 2007 thru 2008 ].
Also the moving from apache to .net happened quite a long time after this in early 2007 thru 2008. Not all the sites have gone over yet.
I have tried to summarise the calendar of events below:
Calendar of Events
1) Multiple sites created between 2004 - 2006 with same content. Canonical issues exist ( on apache )
2) Duplicate content filters kick in within 6 months of creation due to the above issues.
3) Further multiple sites created for different geographies / TLD's , however all with the same content eg. co.uk, com.au etc (on apache)
4) 2007 thru 2008 later multiple sites [ regional TLD's and surplus .COM's ] were modified so they only contained one version of geography specific content – to remove dupe content issues – geographic specific content was then redirected from all sites to the specific geographic site e.g UK content on the all domains was redirected to the .co.uk site, AU content on all domains was redirect to the .com.au site (all sites still on apache and all 301 redirects performed with htaccess)
Part of this change involved redirects for canonical purposes [ e.g. .htm to "/" , + underscores to hyphens , plus URL re writes [ lower case to upper and lower etc ).
5) Surplus sites not required due to duplicate content were redirected either to the home page of our .COM site or the corresponding URL of the .COM site.
site2.com/widget1/ to site1.com/widget1/ and so on through the 30 -40k of URL's
6) Filter on .com site remained in place until now with only an improvement of SERP's appearing in deep positions. The removal of canonical issues brought .COM back to these deep positions - albeit a bit delayed
7) [ in late 2007 ] the .COM site was moved from Apache to .NET – base category URLs remained the same so no redirects were applied here, page names at the product level changed , underscores were replaced with hypens at the product category levels, and individual 301s were set up on all these to redirect to the new product and category pages
8) Filter on .com still seems to be in place but other sites are now working OK after 3-6 months
9) 12-18 months down the track the .com is still filtered , however the .co.uk appears to have fallen into the same filter, perhaps triggered by a high PR IBL and large scale link campaigns , other geographic specific URLs are fine.
- No loops appear to exist
- Duplicate content issues are largely under control - although there appear to be ongoing "issues" cropping up which only affect 2-3% of the total URL's [ as reported in WMT ].
- The potential mis-management of the 301's may caused the .COM site to engage a penalty filter.
I hope this better outlines things for an easier response [ again my apologies ]
| 3:57 am on Aug 20, 2008 (gmt 0)|
|CainIV I don't believe consolidating the sites is the issue |
Nor did I - I believe it is the chain of 301's - the methodology behind trying to save those urls - that is.
|5) Surplus sites not required due to duplicate content were redirected either to the home page of our .COM site or the corresponding URL of the .COM site. |
Whitey - how many domains are 301 redirecting to the .com at this point?
| 4:41 am on Aug 20, 2008 (gmt 0)|
4-5 entire sites to home page
2-3 partial site redirects [ that is where only some of the content / URL's have redirected to .COM ] - but still extensive , say around 15- 30k of page / URL's
| 5:12 am on Aug 20, 2008 (gmt 0)|
> Google is well aware of various types of abuse, and has slapped several well known people for such abuse in recent months. I'm not mentioning those tricks here, except for the one that Matt Cutts blogged about. That was all to do with buying a domain name and existing site, and then redirecting it to another site, purely to try to enhance the PageRank of the site being redirected to.
That type of cheesy scheming deserves to be punished. You can't expect to get away with it. Sounds like what somebody would be in 2001 when it was cool to catch dropped domains with high PR or many IBLs.
| 5:43 am on Aug 20, 2008 (gmt 0)|
|Google is well aware of various types of abuse |
Sure , but this isn't an abuse issue , it's a management practice gone wrong issue, and I'm not sure of how many folks [ most of all myself ] failed to realise how they might fall into it.
| 1:21 pm on Aug 20, 2008 (gmt 0)|
|Sure , but this isn't an abuse issue, it's a management practice gone wrong issue. |
Same thing, they can be synonymous with one another, happens all the time. ;)
| 1:28 pm on Aug 20, 2008 (gmt 0)|
Whitey no offense here but your SEO/management screwed up big time here. There are so many issue here it is impossible to determine what caused the filter and or filters.
Just bad management decisions being made without proper knowledge of the outcome.
| 1:45 pm on Aug 20, 2008 (gmt 0)|
|Whitey no offense here but your SEO/management screwed up big time here. |
I read through all six pages of this thread and then countless other pages in more threads and think, whew, just shut everything down and start over again. It sounds like you've buried yourself so deep within a technical cluster you know what, that there ain't no light at the end of the tunnel. I read the above and "I know" why you are where you're at today. That is an absolute mess you've gotten yourself into.
Isn't it great that you can come to WebmasterWorld and get all this free advice? I'd say there was a good $50,000+ in consulting advice given so far. How much are you paying your current SEO team? Maybe you could get them a subscription to WebmasterWorld? Or, are they here already lurking and acting on our advice?
And, to top all the above off, you are in an industry that has come under heavy fire from Google over the past 12-18 months. There is so much going on that I don't even think a team of the best professionals could bail you out. Even if they did, you've still got another 6, 12, 18 months of recovery. By that time, the Internet will have changed again. :(
Stop wasting your time! Start over again. With just one domain, fresh, no redirects. You'll probably do much better that way then trying to undo what has taken years to do. That would be my "knee-jerk" advice at this point.
| 5:51 pm on Aug 20, 2008 (gmt 0)|
Hi guys. Whitey has shared some information with me about this process. Not so long ago, page helped me with a serious issue that was completely technical on my website that solved numerous indexing problems, so let's work to help Whitey with his. Whitey invited me to share this information with the thread both as a learning tool and for continued help.
This, from a small convo Whitey and I had.
Hi Whitey. While some websites may be perfectly fine with network links, others may not. Incestual linking has nothing to do with the content or difference in content types between two websites. It has everything to do with filters applied to websites when the same websites are on the same network, with the same domain owner and they are all interlinked. All of the search engines can 'see' the link graph for yoursite.com.
Even if you are unfiltered from fixing the 301's, I believe the filter for interlinking will still exist.
One issue here is that geographic-centric information on one domain that belonged on another should not have been redirected, it probably should have been moved, especially in the case of thousands of pages.
I do not believe you need those 301 redirected urls and believe they are hurting you. It is unlikely that all of those inner pages had deep links, but you would know best. If not, it is best for you at this point to look at the entire 301 structure and being getting closer to stability. In a nutshell I would look at removing
For the index.html of index.aspx at yoursite.com is it much better to 301 redirect that to the domain root and not 404 it. It is important that crawlers still are able to access the root document, then move all properties of that document to the canonical root.
Bottom line, is I suggested that the 301 campaign needs to end, he needs to disengage all 301's to the main site and removal interlinking between the sites (all domains are registered as same owner, and most sites are same c block from what i gather) Only then will he really be able to get to the heart of what is wrong.
| 7:06 pm on Aug 20, 2008 (gmt 0)|
|Bottom line, is I suggested that the 301 campaign needs to end, he needs to disengage all 301's to the main site and removal interlinking between the sites (all domains are registered as same owner, and most sites are same c block from what i gather) Only then will he really be able to get to the heart of what is wrong. |
A total cluster! Whitey, how did you get yourself into this? I don't want to ask who recommended that you do all those 301s like that, I really don't. :(
In short, you've been 301'ing all the junk to the .com. Stop the 301s and start over. Just kill all the external juice you've been redirecting. It has actually turned into a toxic mixture and can no longer be classifed as "juice".
You have way too much "junk in your trunk" at this point in time. And, that industry you are in is going to cause you to turn prematurely gray. :)
Whitey, were all these topics somewhat related to your challenges?
|301 redirects - handle with care or be penalised in Google Search News |
Yo-Yo Effect - Observations and Understandings in Google Search News
301 redirects - is Yahoo reliable in Yahoo Search Engine and Directory
Bad Link Campaigns - How to recover and resubmit request in Google Search News
Google TBPR - Grey / Gray - what does it really mean? in Google Search News
Ranking lost on Google UK - but only for English results in Google Search News
Thousands of URLs not indexed - What to do? in Google Search News
Duplicate content - what proportion is dangerous to your site? in Google Search News
WMT - Web crawl glitch in Google Search News
Minus 100 Penalty - for aggressive link exchange? in Google Search News
The "Minus Thirty" Penalty - part 7 in Google Search News
Penalties - forgiven but not forgotten? in Google Search News
UK Google SERP Changes - August 2008 in Google Search News
Matt Cutts: "Some old penalties to go" - which ones? in Google Search News
Linking problems and how to find good links in Link Development
Spotting a manual review in Google Search News
Putting two related sites in the same WMT account in Google Search News
Newly Acquired Domain Penalized - will not rank even after clean-up in Google Search News
| 11:32 pm on Aug 20, 2008 (gmt 0)|
Thanks Guys .... i appreciate your passion .To pinch a little philosophy [ Confucious says ] - " What you know, you know, what you don't know, you don't know. This is true wisdom. "
I'm not sure where Google fits into this with regards to many of us performing and understanding it's inner workings. Google doesn't provide all the answers. But clearly there's a need for us to come together to share bits of knowledge otherwise these threads and others wouldn't exist.
|While some websites may be perfectly fine with network links, others may not. Incestual linking has nothing to do with the content or difference in content types between two websites. It has everything to do with filters applied to websites when the same websites are on the same network, with the same domain owner and they are all interlinked. All of the search engines can 'see' the link graph for yoursite.com. |
Not so sure about this as the sites that are interlinked have different content - but I'm just questioning it for the purposes of debate.
|Adam_Lasnik #:3727434 [webmasterworld.com...] - Internal / cross-domain linking... don't go hog wild. There's no magic threshold. However, penalties for country-domain cross-linking isn't something I have seen. |
I think Adam clarified that cross linking sites of same ownership with different content has not shown itself as an issue - not to say that we haven't become the first. There is no intent beyond what you would see in a normal internal linking situation for sites that rank well and help visitors navigate better to blocks of content that are relevant to their query.
Moved , not redirected
|One issue here is that geographic-centric information on one domain that belonged on another should not have been redirected, it probably should have been moved, especially in the case of thousands of pages. |
Sounds highly plausable - but i think it's when chains, or large quantaties become involved it breaks down. You see we did 301 redirects "once" from an unaffected site to a new domain. The site indexed and ranked within 4 weeks and has been stable for a year.
So it can work.
However, re directing off of the .COM site has been a nightmare
In a nutshell I would look at removing
Yes, surely simplification and getting things back to basics is important.
Surely Google holds a DB of old URL's and therefore must Google be informed [ perhaps by the WMT URL removal tool ] of redundant URL's ? Is this possible ? Any other way ?
|For the index.html of index.aspx at yoursite.com is it much better to 301 redirect that to the domain root and not 404 it. It is important that crawlers still are able to access the root document, then move all properties of that document to the canonical root. |
Any comments to back this up .... it's a fairly definitive statement ?
|Bottom line, is I suggested that the 301 campaign needs to end, he needs to disengage all 301's to the main site and removal interlinking between the sites (all domains are registered as same owner, and most sites are same c block from what i gather) Only then will he really be able to get to the heart of what is wrong |
I want to [ politely ] challenge these statements intended to help :
Same C Block - Same ownership // Links and redirects between sites
Matt Cutts : [mattcutts.com...]
It doesn't seem to be an issue here with several mentions of the same IP and ownership involved , and I'm sure I've seen other clarifying posts from Matt Cutts and Adam Lasnik talking about links between sites and multiple same ownership sites not being an issue.
There's some loose talk about "500 domains" being a problem , but this is about a small number of domains [ with lots of page / URL's though ].
Correct me if I'm wrong
but switching the 301's off - this most certainly sounds the way to go, but how to notify Google and make sure it gets the message ?
Tedster's original post said :
|An honest slip-up or technical confusion that accidentally fits a spammer footprint can be remedied, eventually, through a clean-up and re-inclusion request. |
Would you not use the WMT URL removal tool as well ?
[edited by: Whitey at 11:48 pm (utc) on Aug. 20, 2008]
| This 59 message thread spans 2 pages: < < 59 ( 1  ) |