.htaccess redirect issue - robots.txt

robots.txt issue with redirect

[tommydamic68]

Hello all,

Hoping someone can help with this issue with my .htaccess redirect. So a bit of history, I had a Vbulletin (3.8) forum script that had the .htaaccess file you see below, I was using VBSEO (an SEO script that change the URL to be search engine friendly.) I then moved my forum over to Xenforo software, so the threads needed to be redirected as well as my URL - here is the additional code I was given which I just dropped in the top of the .htaccess file.

Options -Indexes
RewriteEngine On
RewriteCond %{HTTP_HOST} www.mysite.com$
RewriteRule ^(.*)$ http://mysite.com/$1 [R=301,L]
RewriteRule ^[^/]+/.+-([0-9]+)/$ /community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ /community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ /community/ [R=301,L]

The problem is this,
when looking for www.mysite.com/robots.txt it redirects back to www.mysite.com

Please help with this issue, its frustrating to say the least! Also, I would love to be able to rewrite my entire .htaccess file itself with all the extra "stuff" it has gathered over the years and bring it up date with 2014 - perhaps an .htaccess guru could clean this bad boy up!

Thanks in advance!

Tom

Options -Indexes
RewriteEngine On
RewriteCond %{HTTP_HOST} www.mysite.com$
RewriteRule ^(.*)$ http://mysite.com/$1 [R=301,L]
RewriteRule ^[^/]+/.+-([0-9]+)/$ /community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ /community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ /community/ [R=301,L]

# Use PHP 5.3
AddType application/x-httpd-php53 .php 

# Ticket PBB-18872701
<IfModule mod_suphp.c>
 suPHP_ConfigPath /home/admin/public_html
 <Files php.ini>
 order allow,deny
 deny from all
 </Files>
</IfModule> 
# Ticket PBB-18872701

DirectoryIndex index.php
#RewriteEngine Off

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

RewriteCond %{HTTP_HOST} ^mysite.com$
RewriteRule ^/?$ "http\:\/\/www\.mysite\.com\/" [R=301,L]

RewriteCond %{HTTP_HOST} ^mysite.com$
RewriteRule ^forum\.php\/?(.*)$ "http\:\/\/www\.mysite\.com\/forum\.php$1" [R=301,L]

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums/)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www\.mysite\.com
#RewriteRule (.*) http://www.mysite.com/$1 [L,R=301]

RewriteRule ^((urllist|sitemap).*\.(xml|txt)(\.gz)?)$ vbseo_sitemap/vbseo_getsitemap.php?sitemap=$1 [L]

RewriteCond %{REQUEST_URI} !(admincpvioc1/|modcpvioc2/|cron|vbseo_sitemap)
RewriteRule ^((archive/)?(.*\.php(/.*)?))$ vbseo.php [L,QSA]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !/(admincpvioc1|modcpvioc2|clientscript|cpstyles|images)/
RewriteRule ^(.+)$ vbseo.php [L,QSA]

<Files 403.shtml>
order allow,deny
allow from all
</Files>

RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [NC]

#RewriteCond %{HTTP_HOST} ^mysite.com$
#RewriteRule ^/?$ "http\:\/\/www\.mysite\.com\/" [R=301,L]

#RewriteCond %{HTTP_HOST} ^mysite.com$
#RewriteRule ^forum\.+\/?(.*)$ "http\:\/\/www\.mysite\.com\/forum\.php$1" [R=301,L]
# Expire images header
ExpiresActive On
ExpiresDefault A0
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpg A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/ico A2592000
ExpiresByType text/css A2592000
ExpiresByType text/javascript A2592000
# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
# Or, compress certain file types by extension:
<files *.html>
SetOutputFilter DEFLATE
</files>

RewriteEngine On
RewriteRule ^Microsoft-Server-ActiveSync$ hgpush/index.php

[tommydamic68]

Sorry, not sure why the editor did not take my bbcode for the code tags...

[phranque]

IMPORTANT: Please Use Example.com For Domain Names in Posts - Apache Web Server forum [webmasterworld.com]

RewriteCond %{HTTP_HOST} www.mysite.com$
RewriteRule ^(.*)$ http://mysite.com/$1 [R=301,L]

this being the most general of your redirects, it should go after all other external redirects.
usually best to write this as:

RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]

RewriteRule ^[^/]+/.+-([0-9]+)/$ /community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ /community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ /community/ [R=301,L]

all these redirects should target the fully qualified url including canonical protocol and hostname:

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ http://example.com/community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ http://example.com/community/ [R=301,L]

RewriteCond %{HTTP_HOST} ^mysite.com$
RewriteRule ^/?$ "http\:\/\/www\.mysite\.com\/" [R=301,L]

in .htaccess context the request will never start with a slash.
also the quotes and backslashes in the target aren't doing anything for you here.

RewriteCond %{HTTP_USER_AGENT} ^Baiduspider.* [NC]

this isn't doing anything.

[tommydamic68]

Thanks for the reply @phranque , I have made the changes you suggested, however since fist posting this, I have found what i think was the problem of not being able to see the robots.txt. (which did not work with your changes.)

Here is what was changed to enable me to see my robots.txt

Also, the other issue is this:

www.example.com/community does not redirect as it should to
example.com/community - I want both to go to example.com if possible or at least example.com/community no matter whats typed in.

I changed this:

RewriteRule ^.*$ /community/ [R=301,L]

To this:

RewriteRule ^$ /community/ [R=301,L]

Here are the changes you suggested:

Options -Indexes
RewriteEngine On

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ http://example.com/community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ http://example.com/community/ [R=301,L] 
RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L] 

# Use PHP 5.3
AddType application/x-httpd-php53 .php 

# Ticket PBB-18872701
<IfModule mod_suphp.c>
 suPHP_ConfigPath /home/admin/public_html
 <Files php.ini>
 order allow,deny
 deny from all
 </Files>
</IfModule> 
# Ticket PBB-18872701

DirectoryIndex index.php
#RewriteEngine Off

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.
RewriteEngine On

# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums/)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www\.example\.com
#RewriteRule (.*) http://www.example.com/$1 [L,R=301]


<Files 403.shtml>
order allow,deny
allow from all
</Files>


#RewriteCond %{HTTP_HOST} ^example.com$
#RewriteRule ^/?$ "http\:\/\/www\.example\.com\/" [R=301,L]

#RewriteCond %{HTTP_HOST} ^example.com$
#RewriteRule ^forum\.+\/?(.*)$"http\:\/\/www\.example\.com\/forum\.php$1" [R=301,L]
# Expire images header
ExpiresActive On
ExpiresDefault A0
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpg A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/ico A2592000
ExpiresByType text/css A2592000
ExpiresByType text/javascript A2592000
# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
# Or, compress certain file types by extension:
<files *.html>
SetOutputFilter DEFLATE
</files>

RewriteEngine On
RewriteRule ^Microsoft-Server-ActiveSync$ hgpush/index.php

[edited by: tommydamic68 at 7:14 pm (utc) on Jan 12, 2014]

[tommydamic68]

what is the proper bbcode for code tags?

[lucy24]

I changed this:

RewriteRule ^.*$ /community/ [R=301,L]

To this:

RewriteRule ^$ /community/ [R=301,L]

Oh, excellent, because that was the rule that really jumped out and hit me in the face. As originally written, it said "send ALL requests of all kinds-- except the ones in the four previously excluded subdirectories-- to /community/". You still need to give the full protocol-plus-hostname in the target. This is not redundant, because any specific rules come before the with/without www redirect.

Won't search engines get confused if all requests for the root are redirected to a subdirectory?

In the message-posting window, click Style Codes (link on the left) to see what's available. WebmasterWorld is a handrolled forum that predates things like php/bb, so its formats are sometimes different from what you're used to. If you use Quick Reply, hit Preview to get the full posting window. You can also select text and then click the appropriate format button.

Anything beginning in http:// will be turned into a clickable link, with full URL concealed, unless you use example.com. Or example.anything-else.

[tommydamic68]

Oh, excellent, because that was the rule that really jumped out and hit me in the face. As originally written, it said "send ALL requests of all kinds-- except the ones in the four previously excluded subdirectories-- to /community/". You still need to give the full protocol-plus-hostname in the target. This is not redundant, because any specific rules come before the with/without www redirect.

Won't search engines get confused if all requests for the root are redirected to a subdirectory?

In the message-posting window, click Style Codes (link on the left) to see what's available. WebmasterWorld is a handrolled forum that predates things like php/bb, so its formats are sometimes different from what you're used to. If you use Quick Reply, hit Preview to get the full posting window. You can also select text and then click the appropriate format button.

Anything beginning in http:// will be turned into a clickable link, with full URL concealed, unless you use example.com. Or example.anything-else.

Thanks - how do i encase code? Also, any suggestions to my issue stated above? oh - I see you guys are quoting code, ok.

[tommydamic68]

How do I get this with a redirect:

www.example.com/community to
example.com/community

the forum files is not in the root folder but are in the community folder. so public_html/community

In other word I get example.com/community no matter what combination of my sites address is typed in the browser.

is this ok for SEO?

[lucy24]

All forms of the hostname should be redirected to a single preferred form. For most people this is only a question of with/without "www.". But some sites may also get requests for the wrong protocol (http vs. https) and maybe even an appended port number. That's why the domain-name-canonicalization redirect should be expressed as a negative:

!^(example\.com)?$

But that's the fallback redirect. It only applies when no preceding redirects have already kicked in. All page-specific redirects should include the full protocol-plus-name in the target, so there is no possibility of multiple redirects on the same request. Like this (assuming the htaccess file is located in the root directory):

RewriteRule ^$ http://example.com/community/ [R=301,L]

In other word I get example.com/community no matter what combination of my sites address is typed in the browser.

Do you mean that you do get, or that you want to get? And by "get" do you mean the page content or the URL?

If you are typing in
http://www.example.com/
and you get redirected (browser's address bar changes) to
http://example.com/community/
then everything is happening correctly.

---

the forum files is not in the root folder but are in the community folder. so public_html/community

What folder is the root in? I mean the /index.html file that people would see if you had no redirects in place. Is it public_html/ or something else?

htaccess directives only work on files along the same physical path. Ordinarily, you should put your htaccess file at the highest point (shallowest directory) that is "seen" by all requests. Details may depend on server configuration.

Example: My shared hosting uses the "userspace" system. (Some others use "primary vs. addon domain", which works a little differently.) That means there is, first, a physical directory

/blahblah/myname/

and inside that one you find

/blahblah/myname/mail/ (or something like that, don't remember)
/blahblah/myname/site1/
/blahblah/myname/site2/
/blahblah/myname/site3/

The htaccess file located in the /myname/ folder is seen by all requests for my sites. The htaccess file located in /myname/site1/ is seen only by requests for site1 dot com.

---

If you wrap your text in [ code ] tags, auto-linking is suppressed. But you should still use example dot something.

[tommydamic68]

Do you mean that you do get, or that you want to get? And by "get" do you mean the page content or the URL?

If you are typing in
http://www.example.com/
and you get redirected (browser's address bar changes) to
http://example.com/community/
then everything is happening correctly.

What is happening is this - when i type example.com or www.example.com i get example.com/community however, i can also type the url direct into the browser www.example.com/community and it be its own url - so i have example.com/community and www.example.com/community shouldn't it default to example.com/community when typing in www.example.com/community - if so, thats what i am trying to achieve.

thank you for your help thus far.

[phranque]

Here are the changes you suggested:

Options -Indexes
RewriteEngine On

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ http://example.com/community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^.*$ http://example.com/community/ [R=301,L]
RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]

the "^.*$", which i didn't notice the first time around is still there.

try this:

RewriteEngine On

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/index.php?threads/$1/ [R=301,L]

RewriteRule ^forum\.php$ http://example.com/community/ [R=301,L]

RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^$ http://example.com/community/ [R=301,L]

RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]

another thing i didn't notice before regarding this directive:

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/index.php?threads/$1/ [R=301,L]

you don't need to specify index.php in your target since you already have a "DirectoryIndex index.php" directive in your configuration.
this would mean using this:

RewriteRule ^[^/]+/.+-([0-9]+)/$ http://example.com/community/?threads/$1/ [R=301,L]

however i'm wondering whether your intention was to make this be an external redirect or an internal rewrite.

[lucy24]

What is happening is this - when i type example.com or www.example.com i get example.com/community

however, i can also type the url direct into the browser www.example.com/community and it be its own url - so i have example.com/community and www.example.com/community shouldn't it default to example.com/community when typing in www.example.com/community - if so, thats what i am trying to achieve.

Oh, I see. You have two unrelated redirects:
#1 the root-to-/community/ redirect
#2 the domain-name-canonicalization redirect

#1 seems to be working properly. If its target includes "http://example.com" then it will always behave as intended.

#2 is not working. That's where the rule phranque quoted above

RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]

comes in. Make sure this rule is in the mod_rewrite section of your htaccess file as the very last redirect. That means: before any rules that end in [L] alone (if there are any), but after everything else.

[tommydamic68]

Ok- I placed the redirect as suggested and it still is an issue - i hope its ok to have my site url - i was tired of typing exaplm.com lol - will remove if needed. Please visit my url to see the issue, perhaps it will be easier to decipher? [snip]

again the issue is:

typing www.example.com/community should lead to example.com/community and vice versa.

THANKS AGAIN for your help and patience!

Tom

RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]

Options -Indexes
RewriteEngine On
RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} www.example.com$
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
RewriteRule ^[^/]+/.+-([0-9]+)/$ /community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ /community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^$ /community/ [R=301,L]

# Use PHP 5.3
AddType application/x-httpd-php53 .php


DirectoryIndex index.php
#RewriteEngine Off

# Comment the following line (add '#' at the beginning)
# to disable mod_rewrite functions.
# Please note: you still need to disable the hack in
# the vBSEO control panel to stop url rewrites.


# Some servers require the Rewritebase directive to be
# enabled (remove '#' at the beginning to activate)
# Please note: when enabled, you must include the path
# to your root vB folder (i.e. RewriteBase /forums/)
#RewriteBase /

#RewriteCond %{HTTP_HOST} !^www\.example\.com
#RewriteRule (.*) http://www.example.com/$1 [L,R=301]


<Files 403.shtml>
order allow,deny
allow from all
</Files>

#RewriteCond %{HTTP_HOST} ^example.com$
#RewriteRule ^/?$ "http\:\/\/www\.example\.com\/" [R=301,L]

#RewriteCond %{HTTP_HOST} ^example.com$
#RewriteRule ^forum\.+\/?(.*)$ "http\:\/\/www\.example\.com\/forum\.php$1" [R=301,L]
# Expire images header
ExpiresActive On
ExpiresDefault A0
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpg A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/ico A2592000
ExpiresByType text/css A2592000
ExpiresByType text/javascript A2592000
# compress text, html, javascript, css, xml:
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
# Or, compress certain file types by extension:
<files *.html>
SetOutputFilter DEFLATE
</files>

[edited by: phranque at 7:45 pm (utc) on Jan 13, 2014]
[edit reason] Please Use example.com [webmasterworld.com] [/edit]

[lucy24]

NOW do you see why WebmasterWorld insists on example.com?

typing www.example.com/community should lead to example.com/community and vice versa.

I don't understand what you mean by "vice versa" here. You obviously don't mean that each form should redirect to the other; that'a an infinite loop.

Incidentally, is it
/community
or
/community/
? I had the impression /community/ was a real, physical directory. Otherwise there would be no reason for the redirect.

[tommydamic68]

It's http://example.com/community/

So my question is- if you type in

www.example.com/community/ it should redirect to example.com/community/ would that be correct? All should default to http://example.com/community/ no
Matter what combination of my sites URL.

Thank you

[tommydamic68]

Sorry missed this part Lucy24

I had the impression /community/ was a real, physical directory. Otherwise there would be no reason for the redirect.

Community is a folder, so its public_html (root)/community

The forum software resides in the community folder.

[phranque]

RewriteCond %{HTTP_HOST} !^(example.com)?$ [NC]
RewriteRule (.*) http://example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} www.example.com$
RewriteRule ^(.*)$ http://example.com/$1 [R=301,L]
RewriteRule ^[^/]+/.+-([0-9]+)/$ /community/index.php?threads/$1/ [R=301,L]
RewriteRule ^forum\.php$ /community/ [R=301,L]
RewriteRule ^(images|dir2|dir3|dir4)/.*$ - [S=1]
RewriteRule ^$ /community/ [R=301,L]

these rules must be ordered from most specific to most general.

[tommydamic68]

This is what what for someone who may need the correct answer.

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.example\.com$ [OR]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^$ http://example.com/community/$1 [R=301,L]

[lucy24]

RewriteCond %{REQUEST_URI} ^/$

What does this line do that wasn't already done by the body of the rule?

[tommydamic68]

Not sure- I paid a freelancer to fix it - it works. :)

[tommydamic68]

Ok - I have most sorted out but a few items - hopefully you guys and lend a helping hand. What I have achieved thus far:

Can now properly get my robots.txt to work.
My site redirects as wanted www.example.com example.com all direct to http://example.com/community. now I have a slight issue redirecting these few lines:

As of now they default to 404

http://www.example.com/index.shtm
http://example.com/index.shtml
http://www.example.com/index.asp
http://example.com/index.asp
http://www.example.com/default.asp
http://example.com/default.asp
http://www.example.com/index.cfm
http://example.com/index.cfm
http://www.example.com/index.pl
http://example.com/index.pl

Here is my current .htaccess file - any suggestions would be welcome, I just added some code I found here and most of these odd redirects like .asp .htm etc are redirecting to the home page as they should? Any and all suggestions or correction would surely be helpful. All seems to be working properly as far as i can tell. thanks in advance.

# Directory Index
DirectoryIndex index.html

# Redirect default.aspx and index.html requests to URL ending / on www
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*(default|index)\.(aspx|html?)\ HTTP
RewriteRule ^(([^/]+/)*)(default|index)\.(aspx|html?)$ http://www.example.com/$1? [R=301,L]

# Redirect .html URL requests to equivalent .aspx URLs on www
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*[^/.]+\.html\ HTTP
RewriteRule ^(([^/]+/)*[^/.]+)\.html$ http://www.example.com/$1.aspx [R=301,L]

# Redirect non-canonical requests to www
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

# Internally rewrite .aspx URL requests to .html file
RewriteRule ^(([^/]+/)*[^/.]+)\.aspx$ /$1.html [L]

#Redirect VBSEO
RewriteEngine on
RewriteRule f[\d]+/.+-([\d]+)/index([\d]+).html showthread.php?t=$1&page=$2 [NC,L]
RewriteRule f[\d]+/.+-([\d]+)/ showthread.php?t=$1 [NC,L]
RewriteRule f([\d]+)/index([\d]+).html forumdisplay.php?f=$1&page=$2 [NC,L]
RewriteRule f([\d]+)/ forumdisplay.php?f=$1 [NC,L]

#redirect Index
RewriteCond %{THE_REQUEST} ^.*/index\.php
RewriteRule ^(.*)index.php$ /$1 [R=301,L]


#ErrorDocument 404 /404.shtml
ErrorDocument 404 /layout1/index.html
ErrorDocument 500 default

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.example\.com$ [OR]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^$ http://example.com/community/$1 [R=301,L]


# Use PHP 5.4
AddType application/x-httpd-php54 .php


DirectoryIndex index.php

[phranque]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*(default|index)\.(aspx|html?)\ HTTP
RewriteRule ^(([^/]+/)*)(default|index)\.(aspx|html?)$ http://www.example.com/$1? [R=301,L]

those urls should either get a 404 (which is probably not incorrect), or you can make those 301 to the containing directory by slightly modifying the regular expressions in this ruleset:

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*(default|index)\.(aspx?|s?html?)\ HTTP
RewriteRule ^(([^/]+/)*)(default|index)\.(aspx?|s?html?)$ http://www.example.com/$1? [R=301,L]

[tommydamic68]

Thanks phranque. Any suggestions on anything else or does it look ok? Any improvements?

[lucy24]

As of now they default to 404

No reason to do anything else, unless you've got scads of links that spell out a bogus name and it's out of your power to change them. The umbrella approach would be something like

RewriteCond %{THE_REQUEST} \.[a-z] [NC]
RewriteRule ^(([^/]+/)*)(?:index|main|default)\.\w+ et cetera with [NC] flag again

where you simply slather on every name you can think of, and redirect them all to $1. RewriteCond varies, of course; the minimalist form is simply "if the request contains a literal period followed by an alphabetic". You need the "alphabetic" part because "HTTP/1\.[01]" is part of the request, so there's always at least one literal period.

[tommydamic68]

Thanks Lucy24.

I removed a lot of the code, added yours and it seemed to work, however, these four remain. Easy fix?

http://www.example.com/index.cfm
http://example.com/index.cfm
http://www.example.com/index.pl
http://example.com/index.pl

Here is the new .htaccess code.

Options -Indexes
#redirect Index
RewriteCond %{THE_REQUEST} ^.*/index\.php
RewriteRule ^(.*)index.php$ /$1 [R=301,L]

RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*(default|index)\.(aspx?|s?html?)\ HTTP
RewriteRule ^(([^/]+/)*)(default|index)\.(aspx?|s?html?)$ http://www.example.com/$1? [R=301,L]

#Redirect VBSEO
RewriteEngine on
RewriteRule f[\d]+/.+-([\d]+)/index([\d]+).html showthread.php?t=$1&page=$2 [NC,L]
RewriteRule f[\d]+/.+-([\d]+)/ showthread.php?t=$1 [NC,L]
RewriteRule f([\d]+)/index([\d]+).html forumdisplay.php?f=$1&page=$2 [NC,L]
RewriteRule f([\d]+)/ forumdisplay.php?f=$1 [NC,L]


#ErrorDocument 404 /404.shtml
ErrorDocument 404 /layout1/index.html
ErrorDocument 500 default

RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.example\.com$ [OR]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^$ http://example.com/community/$1 [R=301,L]


# Use PHP 5.4
AddType application/x-httpd-php54 .php


DirectoryIndex index.php

[lucy24]

\.(aspx?|s?html?

You don't need to be so specific. Once the request has said "index." or "default." anything that comes afterward has to be an extension.

(index|default)\.\w+$

You probably don't even need the closing anchor, unless you have a weird file-and-directory structure. If you don't use an anchor, \.\w alone will do. In Regular Expressions, \w means "word character": alphanumerics and also _ lowline.

I'd collect all the ErrorDocument and AddType and DirectoryIndex business at the beginning of your htaccess, but that's me. Definitely keep all your RewriteRules together; don't interrupt them with bits of other modules. Since each module is an island, overall order of execution isn't affected by arrangement of the htaccess file. Order only matters within any one mod.

[tommydamic68]

Ok thanks everyone who helped, I think I am finally up and running at full steam! . Lucy24 - i adjusted what you suggested, i hope its right with the line up of items that is, please correct if wrong. I also cleaned up other items that no longer needed to be in the file. Here is the new code below.

Can you point me in the direction of proper expired heads and things of that nature to optimize page load via .htaccess or is this still the correct forum.

Thanks,

Tom

# Protect htaccess
<files .htaccess="">
order allow,deny
deny from all
</files>

# Don't show directories
Options -Indexes

# ErrorDocument 404 /404.shtml
ErrorDocument 404 /error/bara-index.html
ErrorDocument 500 default

# Use PHP 5.4
AddType application/x-httpd-php54 .php

# Directory Index
DirectoryIndex index.html index.php

# Redirect VBSEO
RewriteEngine on
RewriteRule f[\d]+/.+-([\d]+)/index([\d]+).html showthread.php?t=$1&page=$2 [NC,L]
RewriteRule f[\d]+/.+-([\d]+)/ showthread.php?t=$1 [NC,L]
RewriteRule f([\d]+)/index([\d]+).html forumdisplay.php?f=$1&page=$2 [NC,L]
RewriteRule f([\d]+)/ forumdisplay.php?f=$1 [NC,L]

# Redirect index.php
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www\.example\.com$ [OR]
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^$ http://example.com/community/$1 [R=301,L]

# Redirect index.aspx|html|php|shtml|cfm|pl
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*(default|index)\.(php?|aspx?|s?html?|cfm?|pl?)\ HTTP
RewriteRule ^(([^/]+/)*)(default|index)\.(php?|aspx?|s?html?|cfm?|pl?)$ http://www.example.com/$1? [R=301,L]
RewriteRule ^Microsoft-Server-ActiveSync$ hgpush/index.php

# Banned IP's
<Files 403.shtml>
order allow,deny
allow from all
</Files>

deny from 71.202.73.121
deny from 71.54.55.37
deny from 71.23.138.88
deny from 209.240.114.5
deny from 24.203.173.212
deny from 184.98.48.33
deny from 68.235.138.222
deny from 74.179.106.144
deny from 71.202.244.106
deny from 96.236.234.3
deny from 99.149.120.208
deny from 50.54.159.180
deny from 75.31.124.21
deny from 24.22.205.44

#The following line is enough for .js and .css
<IfModule mod_deflate.c>
AddOutputFilter DEFLATE js css
AddOutputFilterByType DEFLATE text/plain text/xml application/xhtml+xml text/css application/xml application/rss+xml application/atom_xml application/x-javascript application/x-httpd-php application/x-httpd-fastphp text/html

#The following lines are to avoid bugs with some browsers
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IfModule>

# BEGIN EXPIRES
<IfModule mod_expires.c>
ExpiresActive On
ExpiresDefault "access plus 10 days"
ExpiresByType text/css "access plus 1 week"
ExpiresByType text/plain "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType application/x-javascript "access plus 1 month"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-icon "access plus 1 year"
</IfModule>
# END EXPIRES

# Cache Headers
<IfModule mod_headers.c>
<FilesMatch "\.(js|css|xml|gz)$">
Header append Vary Accept-Encoding
</FilesMatch>
<FilesMatch "\.(ico|jpe?g|png|gif|swf)$">
Header set Cache-Control "public"
</FilesMatch>
<FilesMatch "\.(css)$">
Header set Cache-Control "public"
</FilesMatch>
<FilesMatch "\.(js)$">
Header set Cache-Control "private"
</FilesMatch>
<FilesMatch "\.(x?html?|php)$">
Header set Cache-Control "private, must-revalidate"
</FilesMatch>
</IfModule>

[tommydamic68]

Sorry - I listed some IP's - could you please remove those moderator?

[lucy24]

Well, as long as they're visible:

Put your IP blocks in numerical order. It doesn't matter much when there's less than 20 all told, but it will save your sanity later. I recently changed mine one step further, to put ARIN and RIPE in separate groups; it's just easier for me to keep track of. (I already had an entirely separate list for China.)

Also: There's rarely any point to blocking an exact IP down to the last digit. Look up the ranges; generally they're server farms that can be blocked in their entirety. This doesn't seem to be the case with your specific numbers, but look into expanding UA-based lockouts.

[g1smd]

Make sure the redirects are listed before the rewrites.

Make sure that every rule has the [L] flag (one missing).