Forum Moderators: Robert Charlton & goodroi
We have made some changes to our site which required 301 redirects to many links and I am a little confused at some results from Webmaster Tools that someone might be able to explain.
Old links:
my-site.com/Something_About_Stuff.html
New links:
my-site.com/something-about-stuff.html
All old links that might have had underscores and capital letters are properly redirecting using 301 to the lowercase and dashes.
What I am curious about.... In Webmaster Tools, Google is reporting a large number of duplicates. It is showing BOTH the old style link and the new links for the same page yet if you click on the old link, it properly redirects to the new. I am assuming the reason for this is that Google picked up the new links naturally via the site and just has not yet gone to the old links and discovered the 301.
Since there is no way to get to these old links, as they 301 redirect, I assume I just have to wait it out until it does a full re-index and sees that the old links are in fact 301 and then dump them and give the PR to the new links.
So far we have not seen a shuffle much of the pages but I wanted to ask this question in case there is something else we have missed. As our site is approximately 30,000 pages where a large number are 301 now, this is just a matter of time and I shouldn't panic.
I hope this all makes sense...
Thanks,
Kris
You'll even sometimes find that some of the URLs end up in the '404 pages' list for a short time some months later.
That's what I was kind of thinking.. Just a matter of time. We only just uploaded this on April 1st and Google visited within minutes of the upload. At the moment, the old links are still the ones showing in the SERPs. They have not been replaced by the new. Again, I am assuming that is just a matter of time as well. We didn't expect Google to visit so quickly after upload.
I do see the old and new in the "site:" check but my thinking is that since it is only April 3rd and it started indexing on April 1st, that the old just hasn't been flushed yet.
My fingers are crossed.
Thanks again,
Kris
Crazy...
I am glad I did...
The other kick in the head is that we had a some "permissions" problems when we uploaded. I was not joking when I said Google visited within a matter of minutes. About 10 minutes after upload we noticed some pages that were "Forbidden" so we frantically went to fix them. During that time Googlebot had already gone through a bunch of these pages while we were trying to fix the incorrect permissions! So much preparation and sometimes you just can't win.
Thanks,
Kris
Confirmation is obtained not by clicking the links, but by using a HTTP Header Checker to look at the actual HTTP Status Code returned for this request.
If you're unlucky, you'll see a 302 code returned instead of the 301 that you actually need.
.
*** "permissions" problems ***
They'll retry those Forbidden pages within days and likely continue on as if nothing had happened.
We are now seeing pages removing from the index and a lot of the old pages dropping lower in the index. So far it is really hanging on to these old links which is frustrating. But we do have an example now of one page where it is removed completely.
Do you think it would be wise to add a list of the old links on like an "Old Sitemap" page so that the bot can come see these old links and redirect to the new ones a little more quickly rather than wait for this to figure itself out?
We are getting extremely nervous that it removed that one page I mentioned above because it had the old link in the index and naturally found the new one and has considered it duplicate and removed it.
Not really sure how else you would introduce new links. We were thinking they would happily switch but of course the bot is going to find the new ones naturally through the site and still have the old ones. We have to wait for it to go through the old ones from other resources and that could be very very slow. So now we are thinking and old sitemap to help it along a little smoother.
Any advice is greatly appreciated here. If you think we should just be expecting a disruption in traffic and we shouldn't worry about the negative movement and just sit tight, that would be good to know as well.
Thanks
Yes, it takes patience - especially if you've changed many urls, as I assume you have by moving to all lowercase from mixed case.
Just "stress test" all your urls as you've been doing, looking for anything you can improve, such as redirect chains or incorrect error handling or old urls still published in your source code.
Ok, so we will hold off on adding an "old urls" sitemap. We don't want to cause any further issues. You are correct, we have about 30,000 pages and many of them are now changed from underscores and upper/lowercase to dashes and all lower. A huge chunk of these pages are now 301. So you are probably right, we just need to be patient.
I will continue to stress test the urls regularly. Every one I check that Google webmster tools reports as a "duplicate title tags" or "duplicate meta descriptions", is in fact correctly 301 redirecting per HTTPFox (HTTP Header Checker). I just don't like seeing that sort of error when I know both pages don't exist. But, I am sure Googlebot doesn't know it yet. I just don't want us getting banned or penalized for so-called duplicates. I feared that this morning when I saw one page completely vanish from the [site:domain.com] results. Yesterday it had both versions: www.site.com/My_Page.html and www.site.com/my-page.html. Today, both gone.
I will continue to watch closely and hope that the old urls finally do move to the end of the [site:domain.com] listings. I know it is visiting and seeing both since it is reporting the errors in Webmaster Tools but it really seems to want to keep the old ones at the front of the listings.
*Patience*
I know I shouldn't be complaining or freaking out on the 5th when we only made the change on the 1st but I want to be on top of everything and anything. Just in case we didn't think of something so we can fix it immediately.
Thanks so much for the help and advice.
Kris
I have noticed that one page that I mentioned above has returned in the [site:domain.com] listing. www.site.com/my-page.html is the version that is has returned in listings while the old is gone still. Good! However, it has not moved the old PR to the new url. Can it take a while for the PR to also be transferred? I was thinking once it took the new url, it would also assign the old PR to it since it was 301.
This page has NOT yet returned in the search results as well.
Thanks again,
Kris
Kris
Thanks again
Kris
You might also see the old URL's being dropped out of the index for a week or so once they are re-crawled by Google. This would give you the indication that G has started the redirect process and the old URL's would start to show the cache of respective new URL's after about 3-7 days after getting dropped from the index.
Thank you for further input. We are definitely noticing a handful of old links now (probably a lot more but hard to find in 30+k pages) that have vanished from the [site:www.my-site.com] results. In place are the redirected new ones.
These pages are NOT found yet in the search results for known keyword(s). We know the keyword phrases they used to show up for as they were pretty high traffic pages and they are definitely gone. I am expecting this is because of all the input everyone has informed us about. The bot needed to find the new, get rid of the old, still no PR which is expected now and once it trusts the page is real, it is from us, and it likes it, etc (which I now presume would be days if not weeks), it will place those pages *somewhere* back in the search results.
So far the index page seems to be hanging on for dear life. It did NOT have a redirect and didn't need one so hopefully it will stay put. I keep expecting it to vanish with all of these changes going on below it but my fingers are crossed.
I am partially regretting such a change but we all voted that we wanted to be rid of the underscores and capital letters for a cleaner look. It was extremely sloppy and was causing confusion. They were about 10 year old links that we just never plucked up the courage to clean up. Just so many pages all at once.... *shudder*
I will keep watching and keep my fingers crossed. Thanks again so much for the input. It is helping tremendously as I would be freaking out right about now but this information has helped to keep me calmer than I would be at the moment.
Cheers,
Kris
Same:
1. site:www.my-site.com is really holding on tight to the OLD urls
2. Webmaster Tools showing both URLS as duplicate Titles, etc.
New:
3. Today the home page (PR 6) of the website has vanished for its' two primary keyword phrases (not showing for allinanchor either). I really hope this would be expected as well since there is simply so much change with 301s.
We were really under the impression 301s were supposed to make a changing process a little smoother than this. :(
Kris
i think the problem isn't so much the use of the 301 as the scope of the whole project.
problem is google will see 30,000 "new" pages (the ones w/ the -'s)- so, even though they are 301'd, and you shouldn't get dup content penalized, they are still "new" - anytime you add 30,000 new pages to a site, major trust issues are raised - i would have advised you to implement the url changes over time (maybe 3,000 pages a week for 10 weeks?)
now that it is done, its hard to say whether you should try to back up some or all of the changes. how severe is your traffic loss? if it is terrible, then i would try reversing a test portion of your site (maybe 1 category or tree branch...) and see if traffic to that section recovers - if so, start restoring others - then go back and re-implement the changes over time.
Thanks for the information! We have been watching very very closely these past two weeks and although it looks very scary, it does look as though our 8 months of preparation are working according to plan and according to the comments that have been posted. I was planning on writing up a timeline of results pretty soon in hopes that our experience could someday be helpful to others. Obviously our change is not complete but I am trying to document as much as possible.
That being said, immediately after launch of this project on April 1st, we started seeing fluctuations. Our traffic started to decline and we only saw Google picking up the new links and leaving the old.
On my last post, April 8th, our index page took a nose dive which was really disconcerting. We thought it *might* happen but were naturally hoping it wouldn't happen. It did.
A couple days after that post, we started seeing EXACTLY what was said from the first handful of post replies. Google was very slowly replacing the old links with the new links. Good news!
Yesterday, April 14th, we noticed our index page pop back for our two primary keyword phrases. We actually have about 7 phrases but those 2 were our big traffic ones. They both returned and with one phrase, we saw something we have never seen. We moved up one notch above a competitor we have been below for 10 years. Granted, over that 10 years there were other competitors we slowly caught up to but this one was exciting for us. It's only been there since yesterday so we don't actually believe it will stick but what a sight. ;)
Right now, we believe Google is approximately 1/2 of the way through the changes. Still a decline in traffic but the last few days have been back on the incline for traffic. So far our lowest dip was on April 11th.
So, maybe it would have been wise to do this 3,000 pages a week but at the same time, this is our lowest time of the year for our site so we were also thinking take the hit now before we expect our traffic to rise again. Until we are actually complete and we see what our traffic is like after all this, we won't truly know how we should have approached this.
We are watching like a hawk so we can be ready if we do need to take a complete about face and attempt an approach you have suggested. For now, we are going to push forward with our fingers crossed.
Back to working on my timeline maybe for others and definitely for future reference for us when we decide to make another insane change.
Cheers,
Kris
Kris
P.S. These are really just updates for anyone who plans to make a large 301 change. I am hoping it will help in the future, especially when mission is accomplished and we know how long we took a hit in the end.
Is this something to be expected? Maybe it's finally doing a flush and re-sort? Or could this be a different problem? I am not sure why everything would be going rather smoothly, then suddenly it seems like a de-index.
Any thoughts appreciated.
Thanks
Kris
