Please help avoid duplicate contents, managing sub-domains via .hta

Use .htaaccess and mod_rewrite to handle sub-domains effectively

[sandyk20]

I am running couple of sub-domains via our main domain: [mydomain.in...]
Sub-domains in use:
[forums.mydomain.in...]
[gallery.mydomain.in...]

Here are the contents of my current .htaacess file placed on [mydomain.in:...]

###############################################
Options +Indexes
Options +FollowSymlinks
RewriteEngine on
RewriteRule ^gallery/([^.]+)$ [gallery.mydomain.in...] [R=301,L]
RewriteRule ^forums/([^.]+)$ [forums.mydomain.in...] [R=301,L]
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000$
RewriteRule ^/?(.*)$ [mydomain.in...] [R=301,L,NS]

ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
ErrorDocument 500 /

RewriteCond %{HTTP_HOST} ^mydomain\.in
RewriteRule ^(.*)$ [mydomain.in...] [R=permanent,L]
###############################################
(I am re-directing all the traffic from the domains IP adres to actual domain URL via above code, and also re-directing all the 404 and/or error pages to main URL in above code.)

Here are the contents of my current .htaacess file placed on [forums.mydomain.in:...]
###############################################
Options +Indexes
Options +FollowSymlinks

ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
ErrorDocument 500 /
###############################################

My Problems:
1.
Serps + Guests are able to get to my sub-domains from following URLs as well:
www.mydomain.in/somepage
forums.mydomain.in/somepage

This is creating lot of dupe content, i want to fix this problem, with a 301/permanent re-direct (without loosing any rankings or getting penalties)

2.
I am running a Vbulletin forum software script, any user typing a wrong URL or trying to access any improper location should to be re-directed to the folder root appropriately, cause I lost lot of pages from my forums and main site due to a recent db crash.
I want to achieve following if a user tries to access:

forums.mydomain.in/showthread.php?t=1233 which does not exists (reason got deleted)
should be re-directed properly to forums.mydomain.in

3.
There are many pages indexed in Google and Yahoo via our Site IP Address as well for example:
000.000.000.000/forums/somepage
I want to re-direct such pages to forums.mydomain.in/somepage

4. While trying to re-direct users from 404 URLs on forum, entire .CSS and image paths appear to be invalid, is there a possible solution for re-directing users from 404 pages to root properly?

Please help me fix my problems:

[edited by: jdMorgan at 4:26 am (utc) on Mar. 2, 2006]
[edit reason] Obscured specifics. [/edit]

[jdMorgan]

sandyk20,

Welcome to WebmasterWorld!

A few comments and clean-ups:

ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.maydomain.in
RewriteRule ^(gallery¦forum)/[^.]+$ - [F]
#
# Redirect gallery and forums subdirectories to subdomains
RewriteCond %{HTTP_HOST} ^www\.mydomain\.in
RewriteRule ^gallery/([^.]+)$ http://gallery.mydomain.in/$1/ [R=301,L]
RewriteCond %{HTTP_HOST} ^www\.mydomain\.in
RewriteRule ^forums/([^.]+)$ http://forums.mydomain.in/$1/ [R=301,L]
#
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000 [OR]
RewriteCond %{HTTP_HOST} ^mydomain\.in
RewriteRule (.*) http://www.mydomain.in/$1 [R=permanent,L]

Please, do not use "/" as the ErrorDocument for a 500-Server Error. You will regret that dearly if you ever make an error that affects your home page -- Your server will go into a loop and basically do a meltdown, until your hosting company disables your account.

I suggest that you don't use "/" for any of the other ErrorDocuments either, if you are worried about duplicate content problems. Use these error documents to handle the errors as defined by HTTP/1.x.

400-Bad Request should just say that -- Bad Request. You can use the default Apache error page for that one.

401-Authentication Required -- Provide help for users with lost username/password & link to contact webmaster.

403-Forbidden -- Why give home page access to a user-agent or IP address that you've forbidden? -- This makes no sense.

404-Not Found -- Create an error page that nicely explains that the requested resource cannot be found. Include a link to your home page and a link to your site map if you have one. You can add a 7-to-10-second Meta-refresh to the home page if you really must, but explain that the page will redirect automatically if you do this.

I hope I have understood your set-up. If not, then the new rule above won't work.

If your css and images break when redirecting, then use server-relative or canonical links for those files; don't use page-relative links. I.e., Use <img src="http://www.example.com/logo.gif"> or <img src="/logo.gif"> and not <img src="logo.gif">. Of course, you can avoid this if you use the ErrorDocuments as described above.

I can't help with your vBulletin problem. You'll need to modify the vBulletin settings or modify the vBulletin script itself to generate a 404 if the database entry for a thread is missing. Apache has no knowledge of your database.

Important: Change any broken pipe "¦" characters above to solid pipe characters before use; Posting on this forum modifies the pipe character, making it invalid for use in mod_rewrite.

Jim

[sandyk20]

Hello thanks for helping me, but the same is still not working?

i am able to access my sub-domains using
www.mydomain.in/forums and www.mydomain.in/gallery

its not getting re-directed to
forums.mydomain.in
gallery.mydomain.in

thanks.

[sandyk20]

sorry to bump back the topic, still awaiting a solution for my problem.

Thanks and Regards,

[sandyk20]

hey JdMorgan,
I am able to get my site contents re-directed from
www.mydomain.in/forums/ to forums.mydomain.in
by adding this line in www.mydomain.in/forums/.htaccess file

Redirect /forums [forums.mydomain.in...]

Can you please give me a proper line to be added in /forums/.htaccess file so that all the current indexed content gets re-directed by 301 and permanent?

Please also possibly give me a new file to be added to www.mydomain.in/.htaccess

Thanks and Regards,

[sandyk20]

This are the contents of my .htaacess files which helps me address all of my issues listed above:

1.
[mydomain.in...]

***********************************
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)/[^.]+$ - [F]
#
#Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\forums$
RewriteRule ^/?(.*)$ [forums.mydomain.in...] [R=permanent,L]
RewriteCond %{HTTP_HOST} ^^000\.000\.000\.000\gallery$
RewriteRule ^/?(.*)$ [gallery.mydomain.in...] [R=permanent,L]
#
#Re-Direct users coming to Site via IP Adress to proper location
RewriteCond %{HTTP_HOST} ^^000\.000\.000\.000 [OR]
RewriteCond %{HTTP_HOST} ^mydomain\.in
RewriteRule (.*) [mydomain.in...] [R=permanent,L]
***********************************

2.
[forums.mydomain.in...]

***********************************
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /forums/ to proper subdomain
redirect 301 /forums/ [forums.mydomain.in...]
***********************************

3.
[gallery.mydomain.in...]

***********************************
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /gallery/ to proper subdomain
redirect 301 /gallery/ [gallery.mydomain.in...]
***********************************

Please take your time to re-confirm if all the above rules, settings are correct.

thanks and regards,

[jdMorgan]

Well, you could shorten this part up a bit:

# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)(.*)$ http://%{HTTP_HOST}$1 [R=301,L]
#
# Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\
RewriteRule ^(gallery¦forums)(.*)$ http://$1.mydomain.in$2 [R=301,L]
#
# Re-Direct users coming to Site via IP Adress or non-canonical doamin to proper domain
RewriteCond %{HTTP_HOST} ^(000\.000\.000\.000¦mydomain\.in)
RewriteRule (.*) http://www.mydomain.in/$1 [R=301,L]

Really, the only way to know is to test it.

Replace all broken pipe "¦" characters above with solid pipe characters before use. Posting on this board modifies that character, and use of a broken pipe character in .htaccess will cause a 500-Server error or unexpected pattern-match behaviour.

Jim

[sandyk20]

the code provided my you in message no.7 is not working?

It gives following error message on browser:
*******************************
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [e-mail] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.
*******************************

I have reverted back to .htaccess file from message no.6

Regards,

[sandyk20]

hello jdmorgan,
there only seems to be a problem in this part of the code you have provided:

**************************
# Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\
RewriteRule ^(gallery¦forums)(.*)$ [$1.mydomain.in$2...] [R=301,L]
**************************

Rest everything is working perfect:
This is my current .htaccess file located on
www.mydomain.in/.htaccess

**************************
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)(.*)$ [%{HTTP_HOST}$1...] [R=301,L]
#
#Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\forums$
RewriteRule ^/?(.*)$ [forums.mydomain.in...] [R=301,L]
RewriteCond %{HTTP_HOST} ^^000\.000\.000\.000\gallery$
RewriteRule ^/?(.*)$ [gallery.mydomain.in...] [R=301,L]
#
#Re-Direct users coming to Site via IP Adress to proper location
RewriteCond %{HTTP_HOST} ^(000\.000\.000\.000¦mydomain\.in)
RewriteRule (.*) [mydomain.in...] [R=301,L]
**************************

[jdMorgan]

If the code goes into .htaccess instead of httpd.conf, then you'll need to add slashes:

RewriteRule ^(gallery¦forums)/(.*)$ http://$1.mydomain.in/$2 [R=301,L]

I would have thought you might have been able to find this problem yourself...

Also, remember to replace the broken pipe character with a solid pipe before use.

Jim

[sandyk20]

solution in your above post (message 10) is not working, returns 500 server error.

This is an entry from errorlog.html file under my cpanel.

[Sun Mar 12 08:40:12 2006] [alert] [client 00.00.00.00] /home/mydomain/public_html/.htaccess: RewriteCond: cannot compile regular expression '^000\\.000\\.000\\.000\\'\n

[jdMorgan]

Are you using a plain-text editor to create this file? If not, you should be. Use NotePad or some other simple, plain-text editor.

I did not post that "^000\\.000\\.000\\.000\\'\n" regular expression in message #10, and it is indeeed mal-formed.

Double-slashes and the \n on the end are not part of mod_rewrite regular-expressions.

Jim

[sandyk20]

hi jdmorgan,
i am extremely sorry to still hang you on this topic/problem of mine, this is my current .htaccess file content located on [mydomain.in...]

########################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)(.*)$ [%{HTTP_HOST}$1...] [R=301,L]
#
#Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\
RewriteRule ^(gallery¦forums)/(.*)$ [$1.mydomain.in...] [R=301,L]
#
#Re-Direct users coming to Site via IP Adress to proper location
RewriteCond %{HTTP_HOST} ^(000\.000\.000\.000¦mydomain\.in)
RewriteRule (.*) [mydomain.in...] [R=301,L]
####################3

it gives me following error:
RewriteCond: bad flag delimiters in errorlog

Message Edit:

Hey i further looked on the code and fixed the error by changing this line
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000\
to
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000

Thanks a lot jdmorgan. :)

[sandyk20]

This are the .htaccess files on my domain currently as follows:

########################################
#http://www.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)(.*)$ [%{HTTP_HOST}$1...] [R=301,L]
#
#Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000
RewriteRule ^(gallery¦forums)/(.*)$ [$1.mydomain.in...] [R=301,L]
#
#Re-Direct users coming to Site via IP Adress to proper location
RewriteCond %{HTTP_HOST} ^(000\.000\.000\.000¦mydomain\.in)
RewriteRule (.*) [mydomain.in...] [R=301,L]
########################################

########################################
#http://forums.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /forums/ to proper subdomain
redirect 301 /forums/ [forums.mydomain.in...]
########################################

########################################
#http://gallery.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /gallery/ to proper subdomain
redirect 301 /gallery/ [gallery.mydomain.in...]
########################################

My last few questions:
1. Please check the above codes those are working perfectly fine on my domain currently, if you can further provide some more optimizations if needed please let me know.

2. Will i face any problems in future/present using those .htaccess codes? I dont wish to loose my listings/rankings.

3. I dont want to loose the indexed contents via IP Address (lakhs of pages indexed via IP) and also via improper domain handling.
i.e., www.mydomain.com/forums/ (lakhs of pages indexed via improper domain URL)

I appreciate your time and support for helping me resolve my problems..
Looking forward to be a part of this community for rest of my life........... :-)

[jdMorgan]

Added comments in italics

This are the .htaccess files on my domain currently as follows:

########################################
#http://www.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#

Using your home page for error-handling may result in problems, especially if you pass all 403, 404, and 410 errors to the home page. This has the potential to create massive duplicate-content problems with your site, and allow someone to ruin your rankings by linking to bad URLs on your site. I recommend that you remove the 400 and 401 ErrorDocument directives completely, and then create appropriae error pages for 403, 404, and 410 errors.

#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
# Prevent duplicate content on gallery and forums subdomain
RewriteCond %{HTTP_HOST} ^(gallery¦forums)\.mydomain.in
RewriteRule ^(gallery¦forums)(.*)$ http://%{HTTP_HOST}$1 [R=301,L]
#
#Re-Direct users coming to forums and gallery via IP Address in SERPs index to proper location
RewriteCond %{HTTP_HOST} ^000\.000\.000\.000
RewriteRule ^(gallery¦forums)/(.*)$ http://$1.mydomain.in/$2 [R=301,L]
#
#Re-Direct users coming to Site via IP Adress to proper location
RewriteCond %{HTTP_HOST} ^(000\.000\.000\.000¦mydomain\.in)
RewriteRule (.*) http://www.mydomain.in/$1 [R=301,L]
########################################

########################################
#http://forums.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /forums/ to proper subdomain
redirect 301 /forums/ http://forums.mydomain.in/
#

This Redirect 301 directive will cause a redirection loop. I suggest using this instead:
#
RewriteCond %{HTTP_HOST} ^www\.mydomain\.in
RewriteRule ^forums/(.*)$ http://forums.mydomain.in/$1 [R=301,L]
#

########################################

########################################
#http://gallery.mydomain.in/.htaccess#
########################################
ErrorDocument 400 /
ErrorDocument 401 /
ErrorDocument 403 /
ErrorDocument 404 /
#
Options +Indexes +FollowSymlinks
RewriteEngine on
RewriteBase /
#
#Re-Direct users coming to /gallery/ to proper subdomain
redirect 301 /gallery/ http://gallery.mydomain.in/

Comment on redirection loop above applies to this code. I suggest using this instead:
#
RewriteCond %{HTTP_HOST} ^www\.mydomain\.in
RewriteRule ^gallery/(.*)$ http://gallery.mydomain.in/$1 [R=301,L]
#

########################################

3. I dont want to lose the indexed contents via IP Address (lots of pages indexed via IP) and also via improper domain handling. i.e., www.mydomain.com/forums/ (lots of pages indexed via improper domain URL)

The 301 redirections above will remove all IP-based pages from the search engines, and replace them with the domain-based URL pages. This may take several months, if the search engines don't crawl your site often.

Jim