mod rewrite rules don't validate the leading folders

htaccess,rewrite,url

[hottrout]

For the past number of months Google Webmaster Tools has been showing increasing numbers of Not Found 404 errors in the crawl errors. It is currently at over 2200 errors. It was at 1200 last week.

The 404's are for URL's on my site that have never existed and the page that they were supposedly linked from is no longer available.

The strange bit is that the URL is made up from valid parts of my site but in a combined incorrect order. Let me explain,

One of the invalid URL's looks like this;

mydomain.com/libraries/radio/libraries/Pictures/gamecovers/images.htm

One part of the URL is correct
mydomain.com/libraries/radio/stations.htm

and the other part is also correct
mydomain.com/libraries/Pictures/gamecovers/images.htm

Google seems to be detecting parts of each url and combining them.

I have no idea how this is being created and I thought that it was just a google glitch. I have requested help on the google webmaster forum several times now to no avail. I would appreciate anyones help with this and have also included the main body of my htaccess file (and apologise in advance)


--------------------


RewriteEngine On
DirectoryIndex index.php
IndexIgnore *

#preserve bandwidth for PHP enabled servers
<ifmodule mod_php4.c>
php_value zlib.output_compression 16386
</ifmodule>

#Send example.com to www.example.com
RewriteCond %{HTTP_HOST} !^www\.example\.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

#Send index.php to root url
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
RewriteRule ^(([^/]+/)*)index\.php$ http://www.example.com/$1 [R=301,L]

<Files /error/403.htm>
order allow,deny
allow from all
</Files>
<Limit GET POST>
order allow,deny
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>

ErrorDocument 400 /error/400.htm
ErrorDocument 401 /error/401.htm
ErrorDocument 403 /error/403.htm
ErrorDocument 404 /error/404.htm
ErrorDocument 500 /error/500.htm

############### Set up Caching
Header unset Etag
FileETag None
# Set up caching on favicon for a loong time
<FilesMatch "\.ico$">
Header set Expires "Mon, 20 Apr 2015 23:30:00 GMT"
</FilesMatch>
# Set up caching on files for 4 weeks
<FilesMatch "\.(pdf|mov|wmv|gif|jpg|jpeg|png|swf|js|css)$">
ExpiresDefault A2419200
Header append Cache-Control "public"
</FilesMatch>
# Set up 4 Hour caching on commonly updated files
<FilesMatch "\.(txt|html|htm)$">
ExpiresDefault A14400
Header append Cache-Control "private, proxy-revalidate, must-revalidate"
</FilesMatch>
# Force no caching for dynamic files
<FilesMatch "\.(php)$">
ExpiresDefault A0
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform"
Header set Pragma "no-cache"
</FilesMatch>
############### End Caching

## Ban IP Address
#order allow,deny
#deny from 83.140.180.69
#allow from all

## Hotlinking Sites Redirect to picture
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?theb9 [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?vinylcollective\.proboards\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?giantbomb [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?dvd4arab [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?hackingtheplanetagain\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?montada [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?evolution-network\.ws [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?mandaver\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?taringa\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?jocuri-pc\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?iligan\.comoj\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?assistirfilmesgratis\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?zonaforo\.meristation\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?prime-news\.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gamehall\.uol\.com\.br [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?tusjuegospc\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gratisjuegos\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?todonintendoroms\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?univers-de-mario\.over-blog\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?baixarjogoscompletos\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ hotlinkingpicture.gif [NC,L]

RewriteRule ^(index|mainmenu)\.(html?(.*))$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libarary(\/\'s|\'s|s|\%27s|27s)/(.*)$ http://www.example.com/Libraries/$2 [R=301,NC,L]

RewriteRule ^Libarary$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Emulation/emulators_summary.htm$ http://www.example.com/emulators/index.php [R=301,NC,L]

RewriteRule ^Libraries/Emulation/NES/ROMs/(.*)(\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1$2 [R=301,NC,L]

RewriteRule ^(.*)ROMs_summary.htm(.*)$ http://www.example.com/roms/index.php [R=301,NC,L]

RewriteRule ^(.*)NES_roms_summary(.*)$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara....*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara...ms_summary.htm$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libar...*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/NES/.*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/snes/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/SNES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_64/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/N64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/BIOS_Roms/.*$ http://www.example.com/roms/index.php?folder=BIOS-System-Boot [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_gameboy/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Gameboy-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_boot_disks/.*(\.html?|\.zip|\.exe)$ http://www.example.com/roms/index.php?folder=PC/DOS-Boot-Disk [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_abandonware_dos/.*$ http://www.example.com/roms/index.php?folder=PC/Abandonware-DOS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Coleco/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Coleco [R=301,NC,L]

RewriteRule ^Libraries/Emulation/C64/.*(\.html?|\.zip|\.rar|\.txt)$ http://www.example.com/roms/index.php?folder=Commodore/C64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/commodore_128/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Commodore/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Plus4/.*$ http://www.example.com/roms/index.php?folder=Commodore/Plus4-C16-C116 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/VIC-20/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Commodore/VIC20 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sam_coupe/.*(\.html?|\.7z)$ http://www.example.com/roms/index.php?folder=MGT/Sam-Coupe [R=301,NC,L]

RewriteRule ^Libraries/Emulation/intellivision/.*(\.html?|\.int)$ http://www.example.com/roms/index.php?folder=Mattel/Intellivision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/fairchild_channel_f/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Fairchild/Channel-F [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Vectrex/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=GCE/Vectrex [R=301,NC,L]

RewriteRule ^Libraries/Emulation/oric_atmos_tangerine_telestrat/.*(\.html?|\.7z)$ http://www.example.com/roms/index.php?folder=Tangerine/Oric-1-Atmos [R=301,NC,L]

RewriteRule ^Libraries/Emulation/texas_instruments_ti-994a/.*(\.html?|\.zip|\.dsk|\.bin)$ http://www.example.com/roms/index.php?folder=Texas-Instruments/TI-99-4A [R=301,NC,L]

RewriteRule ^Libraries/Emulation/watara_supervision/.*(\.html?|\.sz)$ http://www.example.com/roms/index.php?folder=Watara/Supervision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/enterprise_128/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Enterprise/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/magnavox_odyssey2/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Magnavox/Odyssey-2 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_megadrive_genesis/.*$ http://www.example.com/roms/index.php?folder=Sega/Megadrive-Genesis [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_computer_3000/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Computer-3000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_1000/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Game-1000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_gear/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Game-Gear [R=301,NC,L]

RewriteRule ^Libraries/Emulation/SegaMasterSystem/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Master-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari2600/.*$ http://www.example.com/roms/index.php?folder=Atari/2600-VCS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari5200/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/5200 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari7800/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/7800 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_jaguar/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/Jaguar [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_lynx/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/Lynx- [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan_color/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/TurboGraphics16/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=NEC/TurboGrafx-16-PC-Engine [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_game_and_watch/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Game-and-Watch [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_famicom/.*(\.html?|\.zip|\.fds)$ http://www.example.com/roms/index.php?folder=Nintendo/Famicom-Disk-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Spectrum/.*(\.html?|\.zip|\.tzx)$ http://www.example.com/roms/index.php?folder=Sinclair/Spectrum [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_virtual_boy/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_archimedes/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_bbcmicro/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/amstrad_cpc/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Amstrad/CPC [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_8bit/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/8bit [R=301,NC,L]

RewriteRule ^Libraries/Emulation/MSX/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]


RewriteRule ^classifieds/.*$ http://www.example.com/index.php [R=301,NC,L]

RewriteRule ^Libraries/Pictures/NESGameCovers/(.*)$ http://www.example.com/game-box-art-covers/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^blog/retroblog.html$ http://www.example.com/blog/index.php [R=301,NC,L]

RewriteRule ^blog/2008(.*)$ http://www.example.com/blog/index.php [R=301,NC,L]

RewriteRule ^blog/2009(.*)$ http://www.example.com/blog/index.php [R=301,NC,L]

RewriteRule ^blog/labels(.*)$ http://www.example.com/blog/index.php [R=301,NC,L]


RewriteRule ^example_donations\.(htm)$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^(.*)/Retro\sRadio/RetroRadio_Main(.*)$ http://www.example.com/retro_radio/RetroRadio_Main.htm [R=301,NC,L]

RewriteRule ^TOC_Disclaimer.htm$ http://www.example.com/disclaimer.htm [R=301,NC,L]

RewriteRule ^Advertisers.htm$ http://www.example.com/advertising/advertisers.htm [R=301,NC,L]

RewriteRule ^example\sDonations.htm$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^Museum/.*$ http://www.example.com/index.php [R=301,NC,L]

[edited by: hottrout at 5:14 pm (utc) on Jun 27, 2011]

[g1smd]

Unreadable. Use example.com to stop the forum auto-linking the URLs.

Add a blank line after each RewriteRule line. The edit post button is under your username.

There are a ton of errors in the code, but I'm not going to pick through it until I can read it. :)

[hottrout]

Changes made as requested. I feel like a school boy in front of the teacher after having copied someones english homework. I await your disapproving frown and angry glare.

I should also add that the site is hosted on a dedicated server.

[g1smd]

Thanks! That's a LOT better, and much more readable. It made the job 100 times easier for me.

Start a new post below.

Move the "deny", "files", "limit", "etag" & "filesmatch", "compression" and "ErrorDocument" rules to be before the RewriteEngine On line, and in the order quoted in this list.

After the RewriteEngine On line you need to group all the redirects together in order from most specific to most general.

The "hotlinking" ruleset will be first, then rules for individual files then for multiple files. The "index" rule will be second last and the "non-www/www" rule last in the list. That means they move from near the top, to right at the end.

Escape all literal periods in RegEx patterns, e.g.

whatever\.html

or similar.

(\.html?|\.zip)$

simplifies to

\.(html?|zip)$

here.

Where you redirect to a named index file, do not specify the index file name in the redirect. End the target URL with a trailing slash. This avoids a double redirect as you have another rule which takes the index filename off again. That rule only needs to run if external links ask for index filenames. Omit the index.php target from the rules here.

Forward slashes do not ever need to be escaped in .htaccess. Ever.

Where a pattern ends in exactly .*$ delete the .*$ to speed up the rule. There's no point in mod_rewrite parsing the rest of the request only to do nothing with the data.

Where a pattern ends in exactly

(.*)$

AND the backreference is not being used in the rule target, delete the

(.*)$

from the pattern to speed up the rule.

Where a pattern begins exactly

^(.*)

use

^(([^/]+/)*)

instead. It runs a LOT faster. The .* is greedy, promiscuous and ambiguous. You want an "exact match" here, not a weedy .* pattern.

Where you use .* in the MIDDLE of a pattern, it is completely the wrong thing to do. Where that is supposed to match the "filename" part of the URL, use ([^/.]+) instead. This says "read everything until the next slash or period". It runs a LOT faster.

Begin each SECTION of the file with a plain text # comment describing what the next few lines do.

[hottrout]

OK I have made many of the changes you suggested as follows :-

------------------

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
<Files /error/403.htm>
order allow,deny
allow from all
</Files>
<Limit GET POST>
order allow,deny
allow from all
</Limit>

### Ban IP Address
#order allow,deny
#deny from 83.140.180.69
#allow from all

Header unset Etag
FileETag None

### Set up Caching
# Set up caching on favicon for a loong time
<FilesMatch "\.ico$">
Header set Expires "Mon, 20 Apr 2015 23:30:00 GMT"
</FilesMatch>
# Set up caching on files for 4 weeks
<FilesMatch "\.(pdf|mov|wmv|gif|jpg|jpeg|png|swf|js|css)$">
ExpiresDefault A2419200
Header append Cache-Control "public"
</FilesMatch>
# Set up 4 Hour caching on commonly updated files
<FilesMatch "\.(txt|html|htm)$">
ExpiresDefault A14400
Header append Cache-Control "private, proxy-revalidate, must-revalidate"
</FilesMatch>
# Force no caching for dynamic files
<FilesMatch "\.(php)$">
ExpiresDefault A0
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform"
Header set Pragma "no-cache"
</FilesMatch>
### End Caching

###preserve bandwidth for PHP enabled servers
<ifmodule mod_php4.c>
php_value zlib.output_compression 16386
</ifmodule>

ErrorDocument 400 /error/400.htm
ErrorDocument 401 /error/401.htm
ErrorDocument 403 /error/403.htm
ErrorDocument 404 /error/404.htm
ErrorDocument 500 /error/500.htm

RewriteEngine On

### Hotlinking Sites Redirect to picture
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?theb9 [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?vinylcollective\.proboards\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?giantbomb [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?dvd4arab [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?hackingtheplanetagain\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?montada [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?evolution-network\.ws [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?mandaver\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?taringa\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?jocuri-pc\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?iligan\.comoj\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?assistirfilmesgratis\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?zonaforo\.meristation\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?prime-news\.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gamehall\.uol\.com\.br [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?tusjuegospc\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gratisjuegos\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?todonintendoroms\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?univers-de-mario\.over-blog\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?baixarjogoscompletos\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ hotlinkingpicture.gif [NC,L]


RewriteRule ^(index|mainmenu)\.(html?(.*))$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libarary(/\'s|\'s|s|\%27s|27s)/(.*)$ http://www.example.com/Libraries/$2 [R=301,NC,L]

RewriteRule ^Libarary$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Emulation/emulators_summary.htm$ http://www.example.com/emulators/ [R=301,NC,L]

RewriteRule ^Libraries/Emulation/NES/ROMs/(.*)(\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1$2 [R=301,NC,L]

RewriteRule ^(([^/]+/)*)ROMs_summary.htm(.*)$ http://www.example.com/roms/ [R=301,NC,L]

RewriteRule ^(([^/]+/)*)NES_roms_summary(.*)$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara....*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara...ms_summary.htm$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libar...*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/NES/.*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/snes/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/SNES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_64/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/N64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/BIOS_Roms/.*$ http://www.example.com/roms/index.php?folder=BIOS-System-Boot [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_gameboy/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Gameboy-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_boot_disks/.*(\.html?|\.zip|\.exe)$ http://www.example.com/roms/index.php?folder=PC/DOS-Boot-Disk [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_abandonware_dos/.*$ http://www.example.com/roms/index.php?folder=PC/Abandonware-DOS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Coleco/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Coleco [R=301,NC,L]

RewriteRule ^Libraries/Emulation/C64/.*(\.html?|\.zip|\.rar|\.txt)$ http://www.example.com/roms/index.php?folder=Commodore/C64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/commodore_128/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Commodore/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Plus4/.*$ http://www.example.com/roms/index.php?folder=Commodore/Plus4-C16-C116 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/VIC-20/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Commodore/VIC20 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sam_coupe/.*(\.html?|\.7z)$ http://www.example.com/roms/index.php?folder=MGT/Sam-Coupe [R=301,NC,L]

RewriteRule ^Libraries/Emulation/intellivision/.*(\.html?|\.int)$ http://www.example.com/roms/index.php?folder=Mattel/Intellivision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/fairchild_channel_f/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Fairchild/Channel-F [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Vectrex/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=GCE/Vectrex [R=301,NC,L]

RewriteRule ^Libraries/Emulation/oric_atmos_tangerine_telestrat/.*(\.html?|\.7z)$ http://www.example.com/roms/index.php?folder=Tangerine/Oric-1-Atmos [R=301,NC,L]

RewriteRule ^Libraries/Emulation/texas_instruments_ti-994a/.*(\.html?|\.zip|\.dsk|\.bin)$ http://www.example.com/roms/index.php?folder=Texas-Instruments/TI-99-4A [R=301,NC,L]

RewriteRule ^Libraries/Emulation/watara_supervision/.*(\.html?|\.sz)$ http://www.example.com/roms/index.php?folder=Watara/Supervision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/enterprise_128/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Enterprise/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/magnavox_odyssey2/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Magnavox/Odyssey-2 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_megadrive_genesis/.*$ http://www.example.com/roms/index.php?folder=Sega/Megadrive-Genesis [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_computer_3000/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Computer-3000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_1000/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Game-1000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_gear/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Game-Gear [R=301,NC,L]

RewriteRule ^Libraries/Emulation/SegaMasterSystem/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Sega/Master-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari2600/.*$ http://www.example.com/roms/index.php?folder=Atari/2600-VCS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari5200/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/5200 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari7800/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/7800 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_jaguar/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/Jaguar [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_lynx/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/Lynx- [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan_color/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/TurboGraphics16/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=NEC/TurboGrafx-16-PC-Engine [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_game_and_watch/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Game-and-Watch [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_famicom/.*(\.html?|\.zip|\.fds)$ http://www.example.com/roms/index.php?folder=Nintendo/Famicom-Disk-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Spectrum/.*(\.html?|\.zip|\.tzx)$ http://www.example.com/roms/index.php?folder=Sinclair/Spectrum [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_virtual_boy/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_archimedes/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_bbcmicro/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/amstrad_cpc/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Amstrad/CPC [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_8bit/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=Atari/8bit [R=301,NC,L]

RewriteRule ^Libraries/Emulation/MSX/.*(\.html?|\.zip)$ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]


RewriteRule ^classifieds/.*$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Pictures/NESGameCovers/(.*)$ http://www.example.com/game-box-art-covers/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^blog/retroblog.html$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/2008(.*)$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/2009(.*)$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/labels(.*)$ http://www.example.com/blog/ [R=301,NC,L]


RewriteRule ^example_donations\.(htm)$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^(([^/]+/)*)/Retro\sRadio/RetroRadio_Main(.*)$ http://www.example.com/retro_radio/RetroRadio_Main.htm [R=301,NC,L]

RewriteRule ^TOC_Disclaimer.htm$ http://www.example.com/disclaimer.htm [R=301,NC,L]

RewriteRule ^Advertisers.htm$ http://www.example.com/advertising/advertisers.htm [R=301,NC,L]

RewriteRule ^example\sDonations.htm$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^Museum/.*$ http://www.example.com/ [R=301,NC,L]


DirectoryIndex index.php
IndexIgnore *

###send index.php to root url
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
RewriteRule ^(([^/]+/)*)index\.php$ http://www.example.com/$1 [R=301,L]

###Send example.com to www.example.com
RewriteCond %{HTTP_HOST} !^www\.example\.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

----------------

I do not fully understand what you mean by the following so please guide me on this.

1. Escape all literal periods in RegEx patterns, e.g. whatever\.html or similar.

Can you show me where this is wrong in the code ?

2. Where a pattern ends in exactly .*$ delete the .*$ to speed up the rule.

I am not quite sure what you mean?

3. Where a pattern ends in exactly (.*)$ AND the backreference is not being used in the rule target, delete the (.*)$ to speed up the rule.

Again, I am not sure enough of what you mean.

4. Where a pattern begins exactly ^(.*) use ^(([^/]+/)*) instead. It runs a LOT faster.

I think I have made this amendment correctly, please check for me.

5. Where you use .* in the MIDDLE of a pattern, it is completely the wrong thing to do. Where that is supposed to match the "filename" part of the URL, use ([^/.]+) instead.

Are you refering to this line?

RewriteRule ^Libraries/Emulation/NES/ROMs/(.*)(\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1$2 [R=301,NC,L]

(I am expecting detention over this)

[g1smd]

1. Advertisers.htm$ should be Advertisers\.htm$

2. ^Libraries/Emulation/NES/.*$ should be ^Libraries/Emulation/NES/

3. ^blog/2009(.*)$ should be ^blog/2009

5. ^Libraries/Emulation/NES/ROMs/(.*)(\.zip)$
should be ^Libraries/Emulation/NES/ROMs/([^/.]+\.zip)$

6. (\.html?|\.zip)$ simplifies to \.(html?|zip)$ - adjust file list as appropriate.

Each of those fixes applies in multiple places in the file.

[lucy24]

### Hotlinking Sites Redirect to picture

This section jumped out at me. It could bloat beyond all sense. In general it's better to whitelist your hotlinking section: unless the referer is someone you've pre-approved-- don't forget to include your own site!-- send them all to the picture. Also allow null referrers in the form ^-?$ unless you have a particular reason not to. (Some legitimate ISPs don't send a referrer.) So the structure becomes "referrer is NOT goodguy#1 and is NOT goodguy#2 and is NOT my own site and is NOT blank".

2. Where a pattern ends in exactly .*$ delete the .*$ to speed up the rule.

3. Where a pattern ends in exactly (.*)$ AND the backreference is not being used in the rule target, delete the (.*)$ to speed up the rule.

I am not quite sure what you mean?

As written, the rules say

^.* = "there might be some stuff at the beginning of the string"
and
.*$ = "there might be some more stuff at the end of the string"

Since you're not capturing this extra stuff, there's no reason to mention it at all. It just makes more work for the computer. Leave off the anchor along with its adjoining text, and just start with the part that's significant.

[g1smd]

Remove .*$ and (.*)$ completely and replace ^(.*) with a better pattern as previously mentioned.

[hottrout]

I cant thank everyone enough. Following what you have said I realised that to capture everything after a folder in the URL I did not need to add the (.*)$, I simply needed to end with a trailing slash. This was a revalation and so I have changed my code as you suggested. Please read through this version for me and again make suggestions.

--------------------------

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
<Files /error/403.htm>
order allow,deny
allow from all
</Files>
<Limit GET POST>
order allow,deny
allow from all
</Limit>

### Ban IP Address
#order allow,deny
#deny from 83.140.180.69
#allow from all

Header unset Etag
FileETag None

### Set up Caching
# Set up caching on favicon for a loong time
<FilesMatch "\.ico$">
Header set Expires "Mon, 20 Apr 2015 23:30:00 GMT"
</FilesMatch>
# Set up caching on files for 4 weeks
<FilesMatch "\.(pdf|mov|wmv|gif|jpg|jpeg|png|swf|js|css)$">
ExpiresDefault A2419200
Header append Cache-Control "public"
</FilesMatch>
# Set up 4 Hour caching on commonly updated files
<FilesMatch "\.(txt|html|htm)$">
ExpiresDefault A14400
Header append Cache-Control "private, proxy-revalidate, must-revalidate"
</FilesMatch>
# Force no caching for dynamic files
<FilesMatch "\.(php)$">
ExpiresDefault A0
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform"
Header set Pragma "no-cache"
</FilesMatch>
### End Caching

###preserve bandwidth for PHP enabled servers
<ifmodule mod_php4.c>
php_value zlib.output_compression 16386
</ifmodule>

ErrorDocument 400 /error/400.htm
ErrorDocument 401 /error/401.htm
ErrorDocument 403 /error/403.htm
ErrorDocument 404 /error/404.htm
ErrorDocument 500 /error/500.htm

RewriteEngine On

### Hotlinking Sites Redirect to picture
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?theb9 [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?vinylcollective\.proboards\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?giantbomb [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?dvd4arab [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?hackingtheplanetagain\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?montada [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?evolution-network\.ws [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?mandaver\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?taringa\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?jocuri-pc\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?iligan\.comoj\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?assistirfilmesgratis\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?zonaforo\.meristation\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?prime-news\.info [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gamehall\.uol\.com\.br [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?tusjuegospc\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?gratisjuegos\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?todonintendoroms\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?univers-de-mario\.over-blog\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?baixarjogoscompletos\.net [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?thatguywiththeglasses\.com [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ hotlinkingpicture.gif [NC,L]


RewriteRule ^(index|mainmenu)\.(html?(.*))$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libarary(/\'s|\'s|s|\%27s|27s)/(.*)$ http://www.example.com/Libraries/$2 [R=301,NC,L]

RewriteRule ^Libarary$ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Emulation/emulators_summary.htm$ http://www.example.com/emulators/ [R=301,NC,L]

RewriteRule ^Libraries/Emulation/NES/ROMs/([^/.]+\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1$2 [R=301,NC,L]

RewriteRule ^(([^/]+/)*)ROMs_summary.htm(.*)$ http://www.example.com/roms/ [R=301,NC,L]

RewriteRule ^(([^/]+/)*)NES_roms_summary(.*)$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara....*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libara...ms_summary.htm$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libar...*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]



RewriteRule ^Libraries/Emulation/NES/ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/snes/ http://www.example.com/roms/index.php?folder=Nintendo/SNES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_64/ http://www.example.com/roms/index.php?folder=Nintendo/N64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/BIOS_Roms/ http://www.example.com/roms/index.php?folder=BIOS-System-Boot [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_gameboy/ http://www.example.com/roms/index.php?folder=Nintendo/Gameboy-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_boot_disks/ http://www.example.com/roms/index.php?folder=PC/DOS-Boot-Disk [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_abandonware_dos/ http://www.example.com/roms/index.php?folder=PC/Abandonware-DOS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Coleco/ http://www.example.com/roms/index.php?folder=Coleco [R=301,NC,L]

RewriteRule ^Libraries/Emulation/C64/ http://www.example.com/roms/index.php?folder=Commodore/C64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/commodore_128/ http://www.example.com/roms/index.php?folder=Commodore/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Plus4/ http://www.example.com/roms/index.php?folder=Commodore/Plus4-C16-C116 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/VIC-20/ http://www.example.com/roms/index.php?folder=Commodore/VIC20 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sam_coupe/ http://www.example.com/roms/index.php?folder=MGT/Sam-Coupe [R=301,NC,L]

RewriteRule ^Libraries/Emulation/intellivision/ http://www.example.com/roms/index.php?folder=Mattel/Intellivision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/fairchild_channel_f/ http://www.example.com/roms/index.php?folder=Fairchild/Channel-F [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Vectrex/ http://www.example.com/roms/index.php?folder=GCE/Vectrex [R=301,NC,L]

RewriteRule ^Libraries/Emulation/oric_atmos_tangerine_telestrat/ http://www.example.com/roms/index.php?folder=Tangerine/Oric-1-Atmos [R=301,NC,L]

RewriteRule ^Libraries/Emulation/texas_instruments_ti-994a/ http://www.example.com/roms/index.php?folder=Texas-Instruments/TI-99-4A [R=301,NC,L]

RewriteRule ^Libraries/Emulation/watara_supervision/ http://www.example.com/roms/index.php?folder=Watara/Supervision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/enterprise_128/ http://www.example.com/roms/index.php?folder=Enterprise/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/magnavox_odyssey2/ http://www.example.com/roms/index.php?folder=Magnavox/Odyssey-2 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_megadrive_genesis/ http://www.example.com/roms/index.php?folder=Sega/Megadrive-Genesis [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_computer_3000/ http://www.example.com/roms/index.php?folder=Sega/Computer-3000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_1000/ http://www.example.com/roms/index.php?folder=Sega/Game-1000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_gear/ http://www.example.com/roms/index.php?folder=Sega/Game-Gear [R=301,NC,L]

RewriteRule ^Libraries/Emulation/SegaMasterSystem/ http://www.example.com/roms/index.php?folder=Sega/Master-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari2600/ http://www.example.com/roms/index.php?folder=Atari/2600-VCS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari5200/ http://www.example.com/roms/index.php?folder=Atari/5200 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari7800/ http://www.example.com/roms/index.php?folder=Atari/7800 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_jaguar/ http://www.example.com/roms/index.php?folder=Atari/Jaguar [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_lynx/ http://www.example.com/roms/index.php?folder=Atari/Lynx- [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan_color/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/TurboGraphics16/ http://www.example.com/roms/index.php?folder=NEC/TurboGrafx-16-PC-Engine [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_game_and_watch/ http://www.example.com/roms/index.php?folder=Nintendo/Game-and-Watch [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_famicom/ http://www.example.com/roms/index.php?folder=Nintendo/Famicom-Disk-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Spectrum/ http://www.example.com/roms/index.php?folder=Sinclair/Spectrum [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_virtual_boy/ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_archimedes/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_bbcmicro/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/amstrad_cpc/ http://www.example.com/roms/index.php?folder=Amstrad/CPC [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_8bit/ http://www.example.com/roms/index.php?folder=Atari/8bit [R=301,NC,L]

RewriteRule ^Libraries/Emulation/MSX/ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]


RewriteRule ^classifieds/ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Pictures/NESGameCovers/ http://www.example.com/game-box-art-covers/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^blog/retroblog.html$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/2008 http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/2009 http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/labels http://www.example.com/blog/ [R=301,NC,L]


RewriteRule ^example_donations\.(htm)$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^(([^/]+/)*)/Retro\sRadio/RetroRadio_Main(.*)$ http://www.example.com/retro_radio/RetroRadio_Main.htm [R=301,NC,L]

RewriteRule ^TOC_Disclaimer.htm$ http://www.example.com/disclaimer.htm [R=301,NC,L]

RewriteRule ^Advertisers\.htm$ http://www.example.com/advertising/advertisers.htm [R=301,NC,L]

RewriteRule ^example\sDonations.htm$ http://www.example.com/example_donations.php [R=301,NC,L]

RewriteRule ^Museum/ http://www.example.com/ [R=301,NC,L]


DirectoryIndex index.php
IndexIgnore *

###send index.php to root url
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
RewriteRule ^(([^/]+/)*)index\.php$ http://www.example.com/$1 [R=301,L]

###Send example.com to www.example.com
RewriteCond %{HTTP_HOST} !^www\.example\.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

[hottrout]

Can I also ask if you think the original htaccess file was indeed the cause of my bizzar URL's crawled by google.

[g1smd]

There's still a large number of errors in the file because you missed fixing some things.

There are also several NEW errors because of extra typos introduced into the new file.

[lucy24]

Er, is it really spelled "libarary"?

You have a vast series of rules beginning with

RewriteRule ^Libraries/Emulation/NES/ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

and ending with

RewriteRule ^Libraries/Emulation/MSX/ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]

Seems as if with a little careful naming you could collapse most of these into just a few rules. For starters, this would be a good time to rename all your real directories in all lower-case letters. A simple rewrite will take care of any legacy requests.*

For example:

RewriteRule ^Libraries/Emulation/nintendo_virtual_boy/ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy

Is just one of many whose pattern is

RewriteRule ^Libraries/Emulation/([a-z]+)_([a-z\d_]+)/ http://www.example.com/roms/index.php?folder=$1/$2 [NC,R=301]

Note the location of _ in the pattern. Here it is not safe to use \w even if your server is OK with it.

Edit: The hyphen is a complication. Oops. Get rid of them ;)

At worst, you can use pipes to group identical patterns:

RewriteRule ^Libraries/Emulation/nintendo_(name1|name2|name3)/ http://www.example.com/roms/index.php?folder=nintendo/$1 [NC,R=301]

* This didn't come out sounding quite the way I intended, but you know what I mean. So be nice to your grandfather.

[hottrout]

There's still a large number of errors in the file because you missed fixing some things.

There are also several NEW errors because of extra typos introduced into the new file.

Could I ask where the NEW errors and the ones I missed are, I am sorry but I really need some help.

[hottrout]

g1smd,

Could I just ask for you to highlight the errors in my re-written file again. I really appreciate the help with this.

[lucy24]

OK, Rule #3: Use parentheses in Regular Expressions only when (a) you are capturing something for later re-use or (b) there is a pattern such as ([-\w]+/) that might occur more than once.

Use anchors only if the pattern has to begin or end with your search string, or if you need to capture the beginning or ending. Otherwise just search for the relevant part. ("Pattern contains such-and-such.")

Collapsing many rules into a few rules has the added advantage that you end up with fewer places to make mistakes. For example:

RewriteRule ^blog/retroblog.html$ http://www.example.com/blog/ [R=301,NC,L]
RewriteRule ^blog/2008 http://www.example.com/blog/ [R=301,NC,L]
RewriteRule ^blog/2009 http://www.example.com/blog/ [R=301,NC,L]
RewriteRule ^blog/labels http://www.example.com/blog/ [R=301,NC,L]

Are there any blogs that are not being rewritten? If not, they all collapse to a single
RewriteRule ^blog/ et cetera.
At an absolute minimum you've got a
^blog/200[89]
in there. And once you've done this, you will notice that your first pattern ends with a non-slash while the second pattern ends with a slash, meaning that the final form will come out with two //.

In the hotlinking section you don't need the
^http://(.+\.)?
part. Each extra thing you capture is a little more work for the computer. And add
(.+\.)
to the list of forms you should never use, since it translates in RegEx-speak to "capture as much as you can-- whoops! backtrack and stop at the last period-- whoops! backtrack again and stop to check for this string!-- and whoops! do the same thing eighteen more times".

If you needed this element at all it would be in the form
([-\w]+\.)
or at worst
([-\dA-Z]+\.) [NC]
But since you're not doing anything with it, you don't need it.

To capture "example.com" alone,
RewriteCond %{HTTP_HOST} ^example\.com [NC]
The !^www\.example\.com wording would also capture things like "ww.example.com" or "xyz.example.com" which would probably not reach your htaccess in the first place.

[g1smd]

You need to match

!^(www\.example\.com)?$

so that you redirect all non-canonical forms that resolve to the filesystem. This includes requests for other sub-domains and requests with appended port numbers.

The additional ( ) ? stops an infinite loop when a HTTP/1.0 request is received. These do not normally send a host header.

[hottrout]

I am reading and trying really hard to understand everything that you are telling me. Can you just confirm that my understanding is correct on a few points?

I want to point the old blog/index.html to blog/index.php, because blog/2008, blog/2009 and blog/labels are folders, I want everything under them to point to blog/index.php as well. Therefore I can not use ^ReWrite blog/ because this will cause a loop for blog/index.php as well. Is this logic correct?

Lucy also you said :-
And once you've done this, you will notice that your first pattern ends with a non-slash while the second pattern ends with a slash, meaning that the final form will come out with two //.

I dont follow what you are talking about here, please explain some more, I really want to understand.

Lucy you say I need,
RewriteCond %{HTTP_HOST} ^example\.com [NC]

but g1smd you say that I need
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC]

which is correct? I know that ! negates the answer but I still cant get my head around the logic. I also dont think i fully understand the importance of ^ and $ within the statement.

Feeling very bumb right now.

[hottrout]

This is puzzling me :-

RewriteCond %{HTTP_HOST} ^example.com [NC]
RewriteRule ^(.*)$ http://www.example.com$1 [L,R=301]

If . means match any character and * means match 0 or more of the previous character then (.*) must mean match anything. ^(.*)$ must then mean to match anything and store it in $1. So why does

http://www.example.com$1

not result in example.comexample.com/

I fear that I may not fully understand the ^ and $ anchors and their use.

[hottrout]

I have worked through all your other points with my htaccess file and have made many changes. I have also tested this live and it is running without a 500 error. I would again appreciate your feedback on the changes and (I hope) simplification of the code.

---------------------

# .htaccess for www.example.com
# Written by HotTrout [2011-07-05]
# Edited by HotTrout [2011-07-05]

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
<Files /error/403.htm>
order allow,deny
allow from all
</Files>
<Limit GET POST>
order allow,deny
allow from all
</Limit>

# Ban IP Address
#order allow,deny
#deny from 83.140.180.69
#allow from all

Header unset Etag
FileETag None

# Set up caching on favicon for a loong time
<FilesMatch "\.ico$">
Header set Expires "Mon, 20 Apr 2015 23:30:00 GMT"
</FilesMatch>
# Set up caching on files for 4 weeks
<FilesMatch "\.(pdf|mov|wmv|gif|jpg|jpeg|png|swf|js|css)$">
ExpiresDefault A2419200
Header append Cache-Control "public"
</FilesMatch>
# Set up 4 Hour caching on commonly updated files
<FilesMatch "\.(txt|html|htm)$">
ExpiresDefault A14400
Header append Cache-Control "private, proxy-revalidate, must-revalidate"
</FilesMatch>
# Force no caching for dynamic files
<FilesMatch "\.(php)$">
ExpiresDefault A0
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform"
Header set Pragma "no-cache"
</FilesMatch>

# Preserve bandwidth for PHP enabled servers
<ifmodule mod_php4.c>
php_value zlib.output_compression 16386
</ifmodule>

# Redirect Error Codes
ErrorDocument 400 /error/400.htm
ErrorDocument 401 /error/401.htm
ErrorDocument 403 /error/403.htm
ErrorDocument 404 /error/404.htm
ErrorDocument 500 /error/500.htm

# Turn the rewrite engine on
RewriteEngine On

# Redirect Hotlinking Sites to picture
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} theb9 [NC,OR]
RewriteCond %{HTTP_REFERER} vinylcollective\.proboards\.com [NC,OR]
RewriteCond %{HTTP_REFERER} giantbomb [NC,OR]
RewriteCond %{HTTP_REFERER} dvd4arab [NC,OR]
RewriteCond %{HTTP_REFERER} hackingtheplanetagain\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} montada [NC,OR]
RewriteCond %{HTTP_REFERER} evolution\-network\.ws [NC,OR]
RewriteCond %{HTTP_REFERER} mandaver\.net [NC,OR]
RewriteCond %{HTTP_REFERER} taringa\.net [NC,OR]
RewriteCond %{HTTP_REFERER} jocuri\-pc\.com [NC,OR]
RewriteCond %{HTTP_REFERER} iligan\.comoj\.com [NC,OR]
RewriteCond %{HTTP_REFERER} assistirfilmesgratis\.com [NC,OR]
RewriteCond %{HTTP_REFERER} zonaforo\.meristation\.com [NC,OR]
RewriteCond %{HTTP_REFERER} prime\-news\.info [NC,OR]
RewriteCond %{HTTP_REFERER} gamehall\.uol\.com\.br [NC,OR]
RewriteCond %{HTTP_REFERER} tusjuegospc\.org [NC,OR]
RewriteCond %{HTTP_REFERER} gratisjuegos\.org [NC,OR]
RewriteCond %{HTTP_REFERER} thatguywiththeglasses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} todonintendoroms\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} univers\-de\-mario\.over\-blog\.com [NC,OR]
RewriteCond %{HTTP_REFERER} baixarjogoscompletos\.net [NC,OR]
RewriteCond %{HTTP_REFERER} thatguywiththeglasses\.com [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ hotlinkingpicture.gif [NC,L]

# Redirect old root index pages
RewriteRule ^index|mainmenu\.html?$ http://www.example.com/ [R=301,NC,L]

# Redirect misspelt libarary folder and subfolders
RewriteRule ^Libarary(/\'s|\'s|s|\%27s|27s)/(.*)$ http://www.example.com/Libraries/$2 [R=301,NC,L]

# Redirect old htm pages to php
RewriteRule ^Libraries/Emulation/emulators\_summary\.htm$ http://www.example.com/emulators/ [R=301,NC,L]

# Capture and redirect outside links to NES Rom zip files
RewriteRule ^Libraries/Emulation/NES/ROMs/([^/.]+\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1 [R=301,NC,L]

#Redirect the nes_roms_summary.htm file to new NES folder in php
RewriteRule NES\_roms\_summary.*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

# After NES is dealt with then redirect all other roms_summary.htm files to general roms folder in php
RewriteRule ROMs\_summary\.htm.*$ http://www.example.com/roms/ [R=301,NC,L]

# Catch all for remaining misspelt libarary folders
RewriteRule ^Libara.*$ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

# Redirect old rom folders to new php
RewriteRule ^Libraries/Emulation/NES/ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/snes/ http://www.example.com/roms/index.php?folder=Nintendo/SNES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo\_64/ http://www.example.com/roms/index.php?folder=Nintendo/N64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/BIOS\_Roms/ http://www.example.com/roms/index.php?folder=BIOS-System-Boot [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo\_gameboy/ http://www.example.com/roms/index.php?folder=Nintendo/Gameboy-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc\_boot\_disks/ http://www.example.com/roms/index.php?folder=PC/DOS-Boot-Disk [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc\_abandonware\_dos/ http://www.example.com/roms/index.php?folder=PC/Abandonware-DOS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Coleco/ http://www.example.com/roms/index.php?folder=Coleco [R=301,NC,L]

RewriteRule ^Libraries/Emulation/C64/ http://www.example.com/roms/index.php?folder=Commodore/C64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/commodore\_128/ http://www.example.com/roms/index.php?folder=Commodore/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Plus4/ http://www.example.com/roms/index.php?folder=Commodore/Plus4-C16-C116 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/VIC\-20/ http://www.example.com/roms/index.php?folder=Commodore/VIC20 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sam\_coupe/ http://www.example.com/roms/index.php?folder=MGT/Sam-Coupe [R=301,NC,L]

RewriteRule ^Libraries/Emulation/intellivision/ http://www.example.com/roms/index.php?folder=Mattel/Intellivision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/fairchild\_channel\_f/ http://www.example.com/roms/index.php?folder=Fairchild/Channel-F [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Vectrex/ http://www.example.com/roms/index.php?folder=GCE/Vectrex [R=301,NC,L]

RewriteRule ^Libraries/Emulation/oric\_atmos\_tangerine\_telestrat/ http://www.example.com/roms/index.php?folder=Tangerine/Oric-1-Atmos [R=301,NC,L]

RewriteRule ^Libraries/Emulation/texas\_instruments\_ti\-994a/ http://www.example.com/roms/index.php?folder=Texas-Instruments/TI-99-4A [R=301,NC,L]

RewriteRule ^Libraries/Emulation/watara\_supervision/ http://www.example.com/roms/index.php?folder=Watara/Supervision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/enterprise\_128/ http://www.example.com/roms/index.php?folder=Enterprise/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/magnavox\_odyssey2/ http://www.example.com/roms/index.php?folder=Magnavox/Odyssey-2 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega\_megadrive\_genesis/ http://www.example.com/roms/index.php?folder=Sega/Megadrive-Genesis [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega\_computer\_3000/ http://www.example.com/roms/index.php?folder=Sega/Computer-3000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega\_game\_1000/ http://www.example.com/roms/index.php?folder=Sega/Game-1000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega\_game\_gear/ http://www.example.com/roms/index.php?folder=Sega/Game-Gear [R=301,NC,L]

RewriteRule ^Libraries/Emulation/SegaMasterSystem/ http://www.example.com/roms/index.php?folder=Sega/Master-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari2600/ http://www.example.com/roms/index.php?folder=Atari/2600-VCS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari5200/ http://www.example.com/roms/index.php?folder=Atari/5200 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari7800/ http://www.example.com/roms/index.php?folder=Atari/7800 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari\_jaguar/ http://www.example.com/roms/index.php?folder=Atari/Jaguar [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari\_lynx/ http://www.example.com/roms/index.php?folder=Atari/Lynx- [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai\_wonderswan/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai\_wonderswan\_color/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/TurboGraphics16/ http://www.example.com/roms/index.php?folder=NEC/TurboGrafx-16-PC-Engine [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo\_game\_and\_watch/ http://www.example.com/roms/index.php?folder=Nintendo/Game-and-Watch [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo\_famicom/ http://www.example.com/roms/index.php?folder=Nintendo/Famicom-Disk-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Spectrum/ http://www.example.com/roms/index.php?folder=Sinclair/Spectrum [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo\_virtual\_boy/ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn\_archimedes/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn\_bbcmicro/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/amstrad\_cpc/ http://www.example.com/roms/index.php?folder=Amstrad/CPC [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari\_8bit/ http://www.example.com/roms/index.php?folder=Atari/8bit [R=301,NC,L]

RewriteRule ^Libraries/Emulation/MSX/ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]


# Additional less important redirects for missing or old files
RewriteRule ^classifieds/ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Pictures/NESGameCovers/ http://www.example.com/game-box-art-covers/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^blog/retroblog\.html$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/200[89]/ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/labels/ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^website(\_|\s)donations\.htm$ http://www.example.com/website_donations.php [R=301,NC,L]

RewriteRule ^Retro\sRadio/.*$ http://www.example.com/retro_radio/RetroRadio_Main.htm [R=301,NC,L]

RewriteRule ^TOC\_Disclaimer\.htm$ http://www.example.com/disclaimer.htm [R=301,NC,L]

RewriteRule ^Advertisers\.htm$ http://www.example.com/advertising/advertisers.htm [R=301,NC,L]

RewriteRule ^Museum/ http://www.example.com/ [R=301,NC,L]


DirectoryIndex index.php
IndexIgnore *

# Send index.php to root url
#RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
#RewriteRule ^(([^/]+/)*)index\.php$ http://www.example.com/$1 [R=301,L]

# Send example.com to www.example.com
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

[lucy24]

Lucy you say I need,
RewriteCond %{HTTP_HOST} ^example\.com [NC]

but g1smd you say that I need
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC]

which is correct?

g1 is. This is a general truth. Call it Rule #0 :( The business about ()? means "the host has to be either exactly 'www.example.com' or entirely blank".

Therefore I can not use ^ReWrite blog/ because this will cause a loop for blog/index.php as well. Is this logic correct?

Yes, you are right. But if you're redirecting everything except that specific file, you can put it into a RewriteCond. (At some point in the process, requests for "directory" get auto-rewritten to "directory/" which in turn gets auto-rewritten to "directory/index.something" -- the range of "somethings" depends on your server, and some will also try "main.something" -- but I don't remember if this happens before or after the request has met .htaccess.)

If . means match any character and * means match 0 or more of the previous character then (.*) must mean match anything. ^(.*)$ must then mean to match anything and store it in $1. So why does

http://www.example.com$1

not result in example.comexample.com/

Because mod_rewrite [httpd.apache.org] does some extra jiggery-pokery with the first part of the url. See under Substitution of Absolute Urls, but here I shifted the bolding:

Never forget that Pattern is applied to a complete URL in per-server configuration files. However, in per-directory configuration files, the per-directory prefix (which always is the same for a specific directory) is automatically removed for the pattern matching and automatically added after the substitution has been done. This feature is essential for many sorts of rewriting - without this, you would always have to match the parent directory which is not always possible.

There is one exception: If a substitution string starts with ``http://'', then the directory prefix will not be added, and an external redirect or proxy throughput (if flag P is used) is forced!

The "per-directory" part may be confusing because you normally think of your site as the whole thing, as opposed to one directory within your site. But from Apache's POV your entire site is a directory. Unless you're actually running your own little server out of your garage, which I think you said you're not.

[g1smd]

Your non-www to www rule needs to do more than just redirect non-www to www. It needs to redirect any non-canonical hostname request to www. Non-canonical hostnames might look like:

example.com
example.com.
example.com:80
www.example.com
www.example.com.
www.example.com:80

The code

RewriteCond %{HTTP_HOST} ^example\.com [NC]

fails to redirect the last two in that list.


When you use

(.*)

you are capturing only the PATH part of the URL request. That's the bit after the domain name and before any appended parameters.

You can see this by using the Live HTTP Headers extension for Firefox where you will see your browser saying:

GET /apache/4331670.htm?page=20&item=50 HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/6.0 (Windows; U; Windows NT 5.8; en-US; rv:3.5) Gecko/20110707 Firefox/7.77

or somesuch. RewriteRule sees only the path part of the URL request. If you need to look at other things you use a RewriteCond and test a server variable: THE_REQUEST sees the whole "request" line; HTTP_HOST sees the only "host" line, while QUERY_STRING sees only the collection of appended parameters.

Hyphens in RegEx patterns (the referrer patterns) do not need to be escaped. The leading .* in the first RewriteRule is redundant.

Underscores never need to be escaped.

When a rules ends with .*$ these characters are redundant and can be removed.

That's all I spotted on a cursory glance at the code.

[hottrout]

I cant thank you both enough for your help and input. I have learnt a great deal about proper syntax and good form within the htaccess file. I know that my current version is significantly better (not perfect I am sure) that the version posted at the top of this page. I even managed to combine a few of the ReWrite rules successfully.

Going back to the original question and indeed the original reason for starting out on this journey. Do you think (looking at my first htaccess file at the top of the thread) that my original htaccess file was creating the malformed URLs that google was findind and reporting? Therefore do you see any reason why the current htaccess would cause any URL malformation?

The New htaccess file follows :-

# .htaccess for www.example.com
# Written by HotTrout [2011-07-05]
# Edited by HotTrout [2011-07-06]

<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
<Files /error/403.htm>
order allow,deny
allow from all
</Files>
<Limit GET POST>
order allow,deny
allow from all
</Limit>

# Ban IP Address
#order allow,deny
#deny from 83.140.180.69
#allow from all

Header unset Etag
FileETag None

# Set up caching on favicon for a loong time
<FilesMatch "\.ico$">
Header set Expires "Mon, 20 Apr 2015 23:30:00 GMT"
</FilesMatch>
# Set up caching on files for 4 weeks
<FilesMatch "\.(pdf|mov|wmv|gif|jpg|jpeg|png|swf|js|css)$">
ExpiresDefault A2419200
Header append Cache-Control "public"
</FilesMatch>
# Set up 4 Hour caching on commonly updated files
<FilesMatch "\.(txt|html|htm)$">
ExpiresDefault A14400
Header append Cache-Control "private, proxy-revalidate, must-revalidate"
</FilesMatch>
# Force no caching for dynamic files
<FilesMatch "\.(php)$">
ExpiresDefault A0
Header set Cache-Control "no-cache, no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform"
Header set Pragma "no-cache"
</FilesMatch>

# Preserve bandwidth for PHP enabled servers
<ifmodule mod_php4.c>
php_value zlib.output_compression 16386
</ifmodule>

# Redirect Error Codes
ErrorDocument 400 /error/400.htm
ErrorDocument 401 /error/401.htm
ErrorDocument 403 /error/403.htm
ErrorDocument 404 /error/404.htm
ErrorDocument 500 /error/500.htm

# Turn the rewrite engine on
RewriteEngine On

# Redirect Hotlinking Sites to picture
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} theb9 [NC,OR]
RewriteCond %{HTTP_REFERER} vinylcollective\.proboards\.com [NC,OR]
RewriteCond %{HTTP_REFERER} giantbomb [NC,OR]
RewriteCond %{HTTP_REFERER} dvd4arab [NC,OR]
RewriteCond %{HTTP_REFERER} hackingtheplanetagain\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} montada [NC,OR]
RewriteCond %{HTTP_REFERER} evolution-network\.ws [NC,OR]
RewriteCond %{HTTP_REFERER} mandaver\.net [NC,OR]
RewriteCond %{HTTP_REFERER} taringa\.net [NC,OR]
RewriteCond %{HTTP_REFERER} jocuri-pc\.com [NC,OR]
RewriteCond %{HTTP_REFERER} iligan\.comoj\.com [NC,OR]
RewriteCond %{HTTP_REFERER} assistirfilmesgratis\.com [NC,OR]
RewriteCond %{HTTP_REFERER} zonaforo\.meristation\.com [NC,OR]
RewriteCond %{HTTP_REFERER} prime-news\.info [NC,OR]
RewriteCond %{HTTP_REFERER} gamehall\.uol\.com\.br [NC,OR]
RewriteCond %{HTTP_REFERER} tusjuegospc\.org [NC,OR]
RewriteCond %{HTTP_REFERER} gratisjuegos\.org [NC,OR]
RewriteCond %{HTTP_REFERER} thatguywiththeglasses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} todonintendoroms\.blogspot\.com [NC,OR]
RewriteCond %{HTTP_REFERER} univers-de-mario\.over-blog\.com [NC,OR]
RewriteCond %{HTTP_REFERER} baixarjogoscompletos\.net [NC,OR]
RewriteCond %{HTTP_REFERER} thatguywiththeglasses\.com [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ hotlinkingpicture.gif [NC,L]

# Redirect old root index pages
RewriteRule ^index|mainmenu\.html?$ http://www.example.com/ [R=301,NC,L]

# Redirect misspelt libarary folder before other rules
RewriteRule ^Libarary(s|\'s|\%27s|27s)/(.*)$ http://www.example.com/Libraries/$2 [R=301,NC,L]

# Redirect old htm pages to php
RewriteRule ^Libraries/Emulation/emulators_summary\.htm$ http://www.example.com/emulators/ [R=301,NC,L]

# Capture and redirect outside links to NES Rom zip files
RewriteRule ^Libraries/Emulation/NES/ROMs/([^/.]+\.zip)$ http://www.example.com/getfile.php?file=roms/Nintendo/NES/USA/$1 [R=301,NC,L]

#Redirect the nes_roms_summary.htm file to new NES folder in php
RewriteRule NES_roms_summary http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

# After NES is dealt with then redirect all other roms_summary.htm files to general roms folder in php
RewriteRule ROMs_summary\.htm http://www.example.com/roms/ [R=301,NC,L]

# Redirect everything within old rom folders to new php
RewriteRule ^Libraries/Emulation/NES/ http://www.example.com/roms/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/snes/ http://www.example.com/roms/index.php?folder=Nintendo/SNES [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_64/ http://www.example.com/roms/index.php?folder=Nintendo/N64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/BIOS_Roms/ http://www.example.com/roms/index.php?folder=BIOS-System-Boot [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_gameboy/ http://www.example.com/roms/index.php?folder=Nintendo/Gameboy-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_boot_disks/ http://www.example.com/roms/index.php?folder=PC/DOS-Boot-Disk [R=301,NC,L]

RewriteRule ^Libraries/Emulation/pc_abandonware_dos/ http://www.example.com/roms/index.php?folder=PC/Abandonware-DOS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Coleco/ http://www.example.com/roms/index.php?folder=Coleco [R=301,NC,L]

RewriteRule ^Libraries/Emulation/C64/ http://www.example.com/roms/index.php?folder=Commodore/C64 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/commodore\128/ http://www.example.com/roms/index.php?folder=Commodore/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Plus4/ http://www.example.com/roms/index.php?folder=Commodore/Plus4-C16-C116 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/VIC-20/ http://www.example.com/roms/index.php?folder=Commodore/VIC20 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sam_coupe/ http://www.example.com/roms/index.php?folder=MGT/Sam-Coupe [R=301,NC,L]

RewriteRule ^Libraries/Emulation/intellivision/ http://www.example.com/roms/index.php?folder=Mattel/Intellivision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/fairchild_channel_f/ http://www.example.com/roms/index.php?folder=Fairchild/Channel-F [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Vectrex/ http://www.example.com/roms/index.php?folder=GCE/Vectrex [R=301,NC,L]

RewriteRule ^Libraries/Emulation/oric_atmos_tangerine_telestrat/ http://www.example.com/roms/index.php?folder=Tangerine/Oric-1-Atmos [R=301,NC,L]

RewriteRule ^Libraries/Emulation/texas_instruments_ti-994a/ http://www.example.com/roms/index.php?folder=Texas-Instruments/TI-99-4A [R=301,NC,L]

RewriteRule ^Libraries/Emulation/watara_supervision/ http://www.example.com/roms/index.php?folder=Watara/Supervision [R=301,NC,L]

RewriteRule ^Libraries/Emulation/enterprise_128/ http://www.example.com/roms/index.php?folder=Enterprise/128 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/magnavox_odyssey2/ http://www.example.com/roms/index.php?folder=Magnavox/Odyssey-2 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_megadrive_genesis/ http://www.example.com/roms/index.php?folder=Sega/Megadrive-Genesis [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_computer_3000/ http://www.example.com/roms/index.php?folder=Sega/Computer-3000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_1000/ http://www.example.com/roms/index.php?folder=Sega/Game-1000 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/sega_game_gear/ http://www.example.com/roms/index.php?folder=Sega/Game-Gear [R=301,NC,L]

RewriteRule ^Libraries/Emulation/SegaMasterSystem/ http://www.example.com/roms/index.php?folder=Sega/Master-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari2600/ http://www.example.com/roms/index.php?folder=Atari/2600-VCS [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari5200/ http://www.example.com/roms/index.php?folder=Atari/5200 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Atari7800/ http://www.example.com/roms/index.php?folder=Atari/7800 [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_jaguar/ http://www.example.com/roms/index.php?folder=Atari/Jaguar [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_lynx/ http://www.example.com/roms/index.php?folder=Atari/Lynx- [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan [R=301,NC,L]

RewriteRule ^Libraries/Emulation/bandai_wonderswan_color/ http://www.example.com/roms/index.php?folder=Bandai/Wonderswan-Color [R=301,NC,L]

RewriteRule ^Libraries/Emulation/TurboGraphics16/ http://www.example.com/roms/index.php?folder=NEC/TurboGrafx-16-PC-Engine [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_game_and_watch/ http://www.example.com/roms/index.php?folder=Nintendo/Game-and-Watch [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_famicom/ http://www.example.com/roms/index.php?folder=Nintendo/Famicom-Disk-System [R=301,NC,L]

RewriteRule ^Libraries/Emulation/Spectrum/ http://www.example.com/roms/index.php?folder=Sinclair/Spectrum [R=301,NC,L]

RewriteRule ^Libraries/Emulation/nintendo_virtual_boy/ http://www.example.com/roms/index.php?folder=Nintendo/Virtual-Boy [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_archimedes/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/acorn_bbcmicro/ http://www.example.com/roms/index.php?folder=Acorn/Archimedes [R=301,NC,L]

RewriteRule ^Libraries/Emulation/amstrad_cpc/ http://www.example.com/roms/index.php?folder=Amstrad/CPC [R=301,NC,L]

RewriteRule ^Libraries/Emulation/atari_8bit/ http://www.example.com/roms/index.php?folder=Atari/8bit [R=301,NC,L]

RewriteRule ^Libraries/Emulation/MSX/ http://www.example.com/roms/index.php?folder=MSX [R=301,NC,L]


# Additional less important redirects for missing or old files
RewriteRule ^classifieds/ http://www.example.com/ [R=301,NC,L]

RewriteRule ^Libraries/Pictures/NESGameCovers/ http://www.example.com/game-box-art-covers/index.php?folder=Nintendo/NES [R=301,NC,L]

RewriteRule ^blog/retroblog\.html$ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/200[89]/ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^blog/labels/ http://www.example.com/blog/ [R=301,NC,L]

RewriteRule ^website(\_|\s)donations\.htm$ http://www.example.com/website_donations.php [R=301,NC,L]

RewriteRule ^Retro\sRadio/ http://www.example.com/retro_radio/RetroRadio_Main.htm [R=301,NC,L]

RewriteRule ^TOC_Disclaimer\.htm$ http://www.example.com/disclaimer.htm [R=301,NC,L]

RewriteRule ^Advertisers\.htm$ http://www.example.com/advertising/advertisers.htm [R=301,NC,L]

RewriteRule ^Museum/ http://www.example.com/ [R=301,NC,L]


DirectoryIndex index.php
IndexIgnore *

# Send index.php to root url
#RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
#RewriteRule ^(([^/]+/)*)index\.php$ http://www.example.com/$1 [R=301,L]

# Send example.com to www.example.com
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$ [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]

[hottrout]

-----------------------

URGENT UPDATE : I have noticed something else about the (now 5000+) malformed urls that appear in google webmaster crawl errors. It would appear that every malformed URL is created from a folder name that has a space in it.

e.g.

http://www.example.com/Latest News/Links/Links/Libarary's/Emulation/acorn_archimedes/roms/retro_radio/RetroRadio_Main.htm

this url is made from the parts of correct urls, however the folder Latest News has a space and after this folder the URL is nonsense.

This might of course be a red herring but I though I would mention it.

-------------------------

[lucy24]

Well, that's another reason not to use spaces anywhere in an url ;)

On the one hand, the user's browser (and all the intervening stops between their server and the end user) might either collapse the spaces into nothing or convert them to %20 or leave them as-is.

On the other hand, spaces have meaning in .htaccess directives, so you have to be rigorous about escaping them.

For example
spaced name unspacedname

would rewrite "spaced" as "name unspacedname", while

spaced\ name unspacedname

would rewrite "spaced name" as "unspacedname".

It may be possible to make a global rewrite/redirect to get rid of any spaces in outdated links.

(\S+)\ +
redirect to $1 [R=301,N]

But the [N] has to be used WITH EXTREME CAUTION because it means "go back to the top and start over again", repeating until the name rinses clean. If you never have more than nine spaces in any url, it may be safer to write a separate rule for each one. You can find or capture packages in the form (\S+\ +)+ but this doesn't get rid of the space; you have to dump each one separately while preserving the non-spaces.

[g1smd]

I am now looking at a site where Google is requesting malformed URLs. The URL formation starts with a valid page, but tacked on the end is part of the URL for an image or part of the URL for a page that was in use a long time ago when the site previously used Wordpress. It's a mystery how this combination is occurring.

[hottrout]

What you are describing g1smd is excatly what I am experiencing. It also seems to have effected my PR somewhat. The NES page that we were talking about here was PR5 for a very long time and is now PR0. It still has over 300 incomming links from opver 240 domains but no PR :(

[hottrout]

Can I ask something else please. Could the following rule result in any file with mainmenu.htm in it being redirected.

# Redirect old root index pages
RewriteRule ^index|mainmenu\.html?$ http://www.example.com/ [R=301,NC,L]

Surly this will only redirect /mainmenu.htm, /mainmenu.html, index.htm and index.html. It should not redirect any other path?

If I type in

http://www.example.com/nonexistentfolder/mainmenu.htm

It also manages to redirect to the root url and does not give a 404 that I expect. How is the rule above capturing this path?

I ask because google malformed URL's are still increasing and now stand at over 7000 :(

[hottrout]

UPDATE:

I changed the expression to the following and it seems to be working correctly but can you explain as to why?

# Redirect old root index pages
RewriteRule ^(index|mainmenu)\.html?$ http://www.example.com/ [R=301,NC,L]

[lucy24]

Pipes have top priority. That means: unless anything is protected with parentheses, the pipes separate everything on the left from everything on the right. So in your original version, you were allowing two possibilities:

^index

and

mainmenu\.html?$

That's why you needed the parentheses.

[hottrout]

That explains a lot. Hopefully that will start to reduce the errors in google webmaster tools now.

I cant thank you enough for your help with this, I have learnt a great deal and also feel that I understand what I have been doing a lot better.

Do you think there is much more I could do to my htaccess to make it smarter, faster or work better?