Changing URL structure - how many pages to 301 redirect

[Buckley]

Hi,

I'm redesigning my shopping cart and moving it to a new platform. I have 1100 products + content pages. The current URL structure is not good.

e.g.

http://www.example.com/c/24954/1/widget-category-100-to-500.html

http://www.example.com/p/1037729/artists-name-name-of-product.html

So i am looking to clean up the URL structure. I did a search to see how many pages are indexed by the search engines and this was the result:

AllTheWeb2140
AltaVista2160
Google 441
Bing 160
Yahoo 2103

Should i be 301 redirecting every page on the site, the 2100+ that Yahoo and alltheweb have indexed, the 441 google have indexed or another amount altogether?

Most of my traffic comes from Google so that is why i am wondering if i need to do all of them or just the 441 in google.

Doing the 301 redirects is going to be costly as there doesn't seem to be a way to do it easily from my current shopping cart system so i don't want to waste money or time doing them if they are not required yet i want to maintain my PR and rankings as well.

What do you guys think is the best option?

Thanks in advance.

[edited by: tedster at 3:42 pm (utc) on Apr 13, 2010]
[edit reason] switch to example.com [/edit]

[tedster]

If you've got an easy pattern to create a few regex rules to point the old URL patterns to the new, then I say do them all by this method. Otherwise analyze for URLs that have good backlinks plus URLs that are common entry pages (either from search or other sources) and just do those.

[pavlovapete]

What operating system?
What web server application?
How much traffic/ server load?
Can you make clean URLs easily enough going forward?

Myself, I'd be looking to do them all if possible.

I'm not sure what overheads are added by redirects (and there are several methods) but I suspect it is not much.

[Buckley]

Hi, thanks for the replies.

Pavlovapete:

The current site runs on an online shopping cart system with a cms style admin/backend that i log onto. It is developed by a specific company (I'm pretty sure i can't put specifics though?).

I've been told that it isn't easy to do 301 redirects and they would need to be done individually and that it would be labour intensive?

I get on average 200 visits per day to the site

[Buckley]

sorry, the last year it is more like 300 visits per day.

cheers.

[pavlovapete]

Hi Buckley,

IMO you must do 301s. The question then becomes "how". If you are on an Apache webserver you can use patterns as tedster suggests, either in your .htaccess file or with mod_rewrite. The IIS webserver has similar functionality if you use the ISAPI rewrite (or similar) component. In a pinch you can use an IIS 404 error hack.

Most of my traffic comes from Google

So you must do everything you can to keep the G happy and not let any existing indexed URLs dissapear.

it would be labour intensive

Doesn't matter - it has to be done :)

HTH

Cheers

[tedster]

not let any existing indexed URLs dissapear

I know I'm in the minority here, but especially on larger sites I often find it best to let the 404 response do its job and let the new link structure (using the new URLs) do its job.

As long as you 301 the important URLs (those with backlinks as well as the high traffic landing pages) this method works well for me in a pinch, where the old URL structure is not at all friendly to rewrite rules of any kind.

Some sites have 7 figures worth of nasty old URLs, and it just doesn't make practical sense to attempt a 301 in every case. This is especially true because the site: operator is having regular seizures and knowing what is indexed and what isn't is like reading tea leaves.

[TheMadScientist]

I've been told that it isn't easy to do 301 redirects and they would need to be done individually and that it would be labour intensive?

Only because the person answering wasn't quite creative enough, IMO... If there are two distinct patterns, personally, I would probably move the redirects to PHP rather than the .htacces, because you have some options that are easier and more readily available with PHP than with mod_rewrite.

1.) You could possibly construct the new URLs 'on the fly' based on the shopping cart information in the database. (It's tougher to explain than it is for me to do, but if you get someone who's qualified they should understand what I'm saying.) *

2.) You have to leave the redirects in place, and if you have 2100 URLs that have to be redirect individually, that's 2100 RewriteRules that have to be processed on every request...

This isn't as bad in the httpd.conf, because it's 'compiled' at startup of the server, but in .htaccess where it's more likely you will have to make the adjustments the file is 'compiled' for EVERY PAGE ACCESSED even if the rules are not used. You can get around some of this processing by placing the .htaccess file in a lower level directory so it's only processed when the pages being redirected are accessed, but personally, I like to only use one .htaccess file, so what I would probably do in this situation is write a PHP file to handle the redirecting, then use mod_rewrite to rewrite access to the URLs being redirected to the redirect.php file, which would then send the visitor to the correct location and will only add one rule to your .htaccess file, which would tell Apache to serve the content from redirect.php when a URL that should be redirected is requested.

* With PHP you have access to both databases if they're different, and it could be (should be?, usually is) possible to access the product in the old one and then construct the URL for the location from the new one... Remember, they both have a means to get the information from the URL for the same products, so it's usually possible to 'work it backward' from one or the other and determine the new URL construct to send visitors to. If you had to you could even modify one of the databases relatively easily (IMO) to make finding and generating the new constructs from the old relatively easy.

[rjwmotor]

I recently did this. The cart I was running had a built in feature that automatically sent 301's to ALL old pages that were requested. Everything went smoothly for us.

I agree with tedster though. We had about 1550 pages where the URL structure changed and if the cart feature hadn't been there we would have just 301'd the important(category, top sellers) pages.

[Buckley]

Thanks guys, really appreciate your help.

rjwmotor, do you mean the old shopping cart (the one you were leaving)had this feature?

As my current cart is an online pay per month cart system developed by a particular company, i can't see how I'd be able to use this kind of feature. They don't offer it, they don't offer any access to the code of the site, only what you can access through their cms style admin back end which is really limited.

e.g. i had to get them to insert the google analytics code as i couldn't access the header!

[TheMadScientist]

Can you give an example of the new URLs, and will the product IDs stay the same or will they change?

An even better question is can you keep the product IDs the same? It does not matter if they are in the new URLs or not, can you keep them the same when you make the change between carts?

Or, can you keep the product names the same?

Is there anything you can keep the same that identifies one product from another that is anywhere on the page or in the URLs and will be the same on the new pages?

If you there is an 'identifiable similarity' anywhere (even if products are re-organized and products appear on pages with products they did not previously appear with, IOW: the entire organization of the pages has changed, but there is an 'identifiable similarity' in each individual product), then someone who knows PHP could redirect everything for you.

IMO there's too many times when people don't know how to do something so they throw their hands up and take the easier way, but IMO there will be some similarity between the content and URLs on the old pages and the content and URLs on the new pages somewhere, and if there is someone who knows how to scrape and use a database could probably (most likely) redirect everything for you...

[edited by: TheMadScientist at 6:35 am (utc) on Apr 19, 2010]

[pavlovapete]

These URLs are your domain right - not the current shopping cart provider's domain?

[Buckley]

yes i own the domain.

[Buckley]

MadScientist: I, or my developer hasn't started building the site yet. I haven't even picked the company to do the job. Have a proposal out on some freelancer sites.

current url's look like this:

http://www.example.com/c/24954/1/widget-category-100-to-500.html (category)

http://www.example.com/p/1037729/artists-name-name-of-product.html (individual product)

Now i guess i could keep all the url's the same but i thought the url structure i currently have might be hurting my search engine optimisation?

Thoughts?

Should i keep them the same like above or bite the bullet and think long term with new SEO friendlier urls?

They are pretty ugly urls with those numbers in them?

[TheMadScientist]

I'll let you read up on the URL structure and decide on what you think, because I've read conflicting opinions on the effect of structure and usually use short, concise, extensionless URLs myself, but there are times when it doesn't make sense or can't really practically be done because of the size of a site and I end up using longer ones than I would normally like to.

Personally, I think the URL structure could probably be overcome from a search engine perspective, but from a visitor perspective IMO they're a bit ugly, so if it was my site I would think about making the change.

At the same time, I would not want to lose the traffic, rankings, bookmarks, links and other 'goodies' that come along with having them the way they are, so I guess my 'short personal answer' is: If I could redirect them and get the structure so that's the way they will stay (really think it through, because Cool URLs Don't Change [w3.org]) then I'd probably do it and get it out of the way, but if I couldn't find a way to redirect them I'd probably leave them alone. (Until I could find a way to do it at least... :)

IMO and In My Experience, there's usually a way to do the redirecting, but sometimes a higher degree of 'creativity' is necessary than others. In some cases you can simply redirect from one to another using a regular expression and in other cases you might have to build a database to do the redirecting at all efficiently. The difference is usually in whether you need to access the new full URL to determine what it is (or build it) from a fragment or not.

EG
http://www.example.com/p/1037729/artists-name-name-of-product.html
to
http://www.example.com/product/1037729

Is a fairly easy association to make...
RewriteRule ^p/([0-9]+)/ http://www.example.com/product/$1 [R=301,L]

This is more difficult, unless there is a consistency in the number of hyphens, which there probably would not be...
http://www.example.com/p/1037729/artists-name-name-of-product.html
to
http://www.example.com/artist-name/name-of-product

The reason is if there are a different number of hyphens you cannot consistently tell where the artist-name stops and the name-of-product starts, so I would probably switch from mod_rewrite to PHP, because it's possible with PHP (and fairly easy, IMO without looking at the site, if you keep the same product numbers in the database when you make the change) where with anything except very advanced mod_rewrite you'd have to do it individually. (It's actually not impossible to do AFAIK with mod_rewrite, but it's more of a 'technical challenge' than it is with PHP IMO.)

Anyway, I'd probably change them right after I found a way to do the redirects, because IMO when thinking about what they look like to visitors 'short and clean' is in, but I'd also want to make sure once they're done that's the way they stay...

As far as 'ugly with the numbers' goes have at look at the URLs here... They used to be all numbers, then they were finally switched to text for the directory names... Maybe so I didn't have to try and remember what forum number this was when I wanted to type a link or go straight here? IDK, but I like them better as text for the directory name and the numbers aren't hurting (and didn't hurt) the rankings any.

[rjwmotor]

"rjwmotor, do you mean the old shopping cart (the one you were leaving)had this feature?"

The cart I run updated to a newer version and they had a built in feature that automatically 301'd the ugly URL's to seo friendly ones. I didn't switch carts; just updated.