lucy24 - 6:02 am on Nov 20, 2012 (gmt 0)

I am pretty sure there are too many to 301 redirect in the .htaccess - my .htaccess would be huge.

For heaven's sake, you don't need to redirect each URL separately :) Depending on how many characters are involved-- how many different ones, and how many in a single URL-- you'll need either a script detour, as above, or just a few lines in htaccess.

But you have a double problem: Somehow you've come out with URLs encoded in 1252 (windows codepage). The 9x range shouldn't occur at all, ever, and the en dash character (not a hyphen) likewise shouldn't occur in an URL.

Redirecting is fine for URLs that already exist. But you also have to do whatever it takes to ensure that nothing in the URL-generating process uses 1252. Fine-tooth-comb your pages-- I assume you're dealing with user-generated content? --and the function that converts content into URLs, and make sure every single step is rigorously locked into the same encoding. UTF-8 is obviously the best choice. And then write a subroutine to intercept any non-ASCII characters before they ever get to be URLs.