Rewriting URLs containing percent-encoded characters

Google and Yahoo adding %09 and %20 in links

[caspita]

Hi all here,

This is what is happenig to my site. For some unknown reason (I have done large research and could not figure it out), GGbot keeps requeting URLs from my site using a 'tab' char in the URLs, something like:

http://www.example.com/section/page%09%09%09.html

of course the real page looks like:

http://www.example.com/section/page.html

I'm already rewriting the URL to a php script and because my content is all in a MySQL DB, of course, when I get the rquest with the '%09' in it I cannot find out the row in the table. I did a fix in the php code to remove the '%09' but then I ended up with dups in the SERPs. So what I want to do better is to redirect these requests to the real URL with no '%09%'s in it.

The number of '%09's is variable, some times 2, 3 or 4, and I have seen not very often 1 or 5 also.

Same think happens with Y!, but Y! is using a %20 instead.

Could you help me please with the simple rule to rewrite the above?

Again, I want to rewrite:

http://www.example.com/section/page%09%09%09.html

to

http://www.example.com/section/page.html

Thanks a lot in advance.
CS.

[jdMorgan]

caspita,

Please review our forum charter [webmasterworld.com]. We ask that all members make an attempt to solve problems on their own, and post their code, so that this forum is clearly seen as a discussion forum, not a help desk. There are simply too many requests for the very small number of volunteers here to write code for everyone who asks.

Because your problem is severe and tricky to fix, and because this problem is apparently becoming widespread, I will post a solution in the hope that it helps several people.

Place this code as close as possible to the top of your other RewriteRules. It can 'call itself' repeatedly and so is very inefficient if placed deep within an .htaccess file.

# If present, delete percent-encoded characters from URL, set 'Fixed' flag
# to 'yes', and restart mod_rewrite in case multiple instances are present.
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^%]*)\%(25)?[A-Z0-9]{2}([^\ ]*)\ HTTP/
RewriteRule .* /%1%3 [E=Fixed:yes,N]
# If 'Fixed' flag is 'yes', do 301 redirect to correct requested URL
RewriteCond %{ENV:Fixed} ^yes$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

Jim

[caspita]

Hi Jim,

Thanks a lot for your help and belive me ;-) I don't use to come here until I feel I'm very lost or when I'm so scare of messing up my system :-). I did try some self-made rules but now that I see your code, God! that I was way more than lost LOL!.

I'll try to understand this thing first because it looks so confusing, I didn't post my code because i was thinking the rule was so easy and I was just getting complicated with REs. But you code just scare me :-)

Thanks again, as always you are a pro!

CS.