Certain redirects cached before others

[Hannahness]

Hi all,

The site that I'm working with implemented 301 redirects across several specific subfolders about a week ago. I've noticed that a bunch of the pages (not sure, percentage-wise, how many) have had the new URL cached, whereas a good portion of them have not. Anyone know why this would be the case?

[tedster]

Sounds like it's the googlebot crawl budget for your site. You may get some insight from the googlebot history in your server logs. It's not uncommon for Google to crawl a 301 several times before trusting it.

One week is pretty good to get even some 301 redirects indexed. To me that sounds like you have done a technically sound job with the site.

[Sgt_Kickaxe]

Give it time, that's normal.

I moved an entire site with several hundred pages from one server/host to another a few months ago and in the process changed every URL to remove "index.php". ie: example.com/index.php/stuff.html which is similar to renaming a folder. The "index.php" was a workaround for pretty permalinks on the old host that refused .htaccess access.

Anyway - the entire site was properly indexed without missing a beat within a month (Google) and 2 months (Yahoo/Bing). Zero traffic loss on any search engine, in fact it's increased 6%-8% over the same period last year on Google, no change on the other two.

Some things I did that may or may not have helped besides setting up a proper redirect include

~ addition of a noarchive meta tag sitewide, removed it when serps displayed no cache available sitewide and all new urls. The goal was to clear the old cache while the 301 propagates.

~ addition of all NEW urls to a sitemap that still had all OLD urls listed. The goal was to ensure serps visited both the new and the old in order to pick up the 301 redirect. The OLD were removed at the same time as the noarchive tag was removed, again when no trace was left of the old links in serps.

Overkill? Maybe, but since the transition went completely smoothly on all major engines I see no reason to avoid taking simple precautions. Only the search engines know if it helped or not.

Keep one other thing in mind, your site has a network of internal and external links telling serps which pages are deemed most important, it takes time for the weight to move from old to new pages via 301 since the site isn't crawled all at one time. Old pages can outrank new pages even after a 301 redirect until that redirect reaches all of the other internal pages pointing to the old url.

tip: make no other major site changes at the same time as trying to redirect old urls to new urls, it probably helps indexing speed/accuracy that the content is identical since 301 suggests it is. Again, maybe not.

[Hannahness]

Thanks Tedster and Sgt_Kickaxe! Informative stuff...we'll see what happens with the non-cached redirected URLs.

[StuffOfInterest]

301 redirects can have issues. I recently renamed one page on my site from the form "AB-Widgets.html" to "Alpha-Bravo-Widgets.html" and left a 301 redirect behind at the old page name. When the old name was in place I was showing up at number two or three for searches on "Alpha Bravo Widgets".

For the first few days after the redirect all traffic was coming in via the old name and then redirecting to the new name. After a few days Google traffic for that page dropped about 50%. My thought at that time was that it was reindexing, so I was calm. A few days later the traffic came back on the new page with no traffic going to the old URL. Great! Wait, not so fast.

Less than four days after the return of the traffic almost all traffic stopped on the the page again. Now it has been over eight days and still no return. If I do a Google search now on "Alpha Bravo Widgets" I find that my page is at position 10. Oops.

End effect is that after about 2-1/2 weeks I've gone from position 3 to position 10 after doing nothing but changing the name of a page and putting a redirect in place. Hopefully my position (and traffic) will return eventually but my case should serve as a warning to be careful when doing renames and 301 redirects as you can't be sure your traffic will preserve through the reindexing.

[tedster]

Thanks, StuffOfInterest. We often talk quite casually about 301 redirects, but it definitely should be noted, and strongly noted, that "Cool URIs Don't Change".

[Hannahness]

Fortunately, the site that I'm working on only hired me now - their rankings aren't great, so hopefully it will only go up from here.