virtual - dynamic subdomains

what about robots.txt and other common support files?

[egyption dev]

hello , that is my first post at this forum
firstly , I apologize for my poor English language

i have create dynamic subdomain system
after some Amendments in htaccess, httpd.conf and server

and it work well
but i want to register at google webmastertool with my subdomains

but what about "robots.txt" and "authentication file"
what i can do in htaccess file to fix this problem
http://dynamic.example.com/robots.txt

my htaccess file ;

RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\.mysite.com
RewriteCond %{HTTP_HOST} ([^.]+)\.mysite.com
RewriteRule ^(.*)$ /home/user/public_html/category.php?slug=%1

[edited by: jdMorgan at 8:27 pm (utc) on Feb. 20, 2009]
[edit reason] example.com [/edit]

[egyption dev]

i have created file "subrobots.txt" in public_html folder
i want to show this file content when the visitor goto http://dynamic.example.com/robots.txt

[edited by: jdMorgan at 8:28 pm (utc) on Feb. 20, 2009]
[edit reason] example.com [/edit]

[jdMorgan]

One way to do it:

RewriteEngine on
#
# Externally redirect to canonicalize the main domain
RewriteCond %{HTTP_HOST} ^example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com:[0-9]+$ [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]
#
# Externally redirect to canonicalize subdomains
RewriteCond %{HTTP_HOST} ^www\.([^.]+)\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.www\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com:[0-9]+ [NC]
RewriteRule (.*) http://%1.example.com/$1 [R=301,L]
#
# Internally rewrite subdomain special file requests to /home/user/public
# folder with "sub-<subdomain>-" prefix on filename. Example:
# foo.example.com/robots.txt --> /home/user/public_html/sub-foo-robots.txt
RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.(www\.)?example\.com
RewriteCond $1 ^robots\.txt$ [OR]
RewriteCond $1 ^sitemap\.xml$ [OR]
RewriteCond $1 ^google[0-9a-f]{16}\.html$ [OR]
RewriteCond $1 ^y_key_[0-9a-z]{16}\.html$ [OR]
RewriteCond $1 ^LiveSearchSiteAuth\.xml$
RewriteRule (.*) /home/user/public_html/sub-%1-$1 [L]
#
# Internally rewrite all other subdomain requests to script
RewriteCond $1 !^sub-
RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.(www\.)?example\.com
RewriteRule (.*) /home/user/public_html/category.php?slug=%1 [L]

Jim

[egyption dev]

hello , thanks for fine code
but i'm sorry ,

when i goto :
[sub.example.com...]
OR
[sub.example.com...]
OR any file after subdoamin url [sub.example.com...]

all urls 's result is :
[sub.example.com...] but no redirect

and i did't get the file's content

sorry , for my language

[egyption dev]

i mean :

[sub.example.com...] = [sub.example.com...]

[jdMorgan]

Note the comments in the code. This method requires you to make some changes to the filenames:

# Internally rewrite subdomain special file requests to /home/user/public
# folder with "sub-<subdomain>-" prefix on filename. Example:
# foo.example.com/robots.txt --> /home/user/public_html/sub-foo-robots.txt

This code *does not* do an external redirect, it does an internal rewrite. The requested URL in your browser address bar will not change.

You can simplify this line

RewriteCond %{HTTP_HOST} ^([^.]+)\.(www\.)?example\.com 

to this

RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com 

but this will not change how the code works. It is just a clean-up that I forgot to do after adding the domain canonicalization rules.

Jim

[egyption dev]

thanks Jim for replay
yes i know that

but when i goto :
[subdomain.example.com...]
i can't see the "sub-subdomain-robots.txt" file's content
but i get :
[subdomain.example.com...] 's content

why the "sub-subdomain-robots.txt" file's content not display

[jdMorgan]

Try changing the rule temporarily (for test purposes only) to

 RewriteRule (.*) http://www.example.com/sub-%1-$1 [R=302,L]

and tell us what you see in your browser address bar after requesting the sub.example.com/robots.txt URL.

If that works, then you may want to try

RewriteRule (.*) /sub-%1-$1 [L]

or some other variations on the rewritten filepath. (It is not clear to me why you need to include the "/home/user/public_html" path in the rewritten filepaths, since this should already be defined as your DocumentRoot. I included it in the code I posted simply because you included it.)

Jim

[egyption dev]

Unfortunately, not working well
after changing the line to :
RewriteRule (.*) http://www.example.com/sub-%1-$1 [R=302,L]
and when i goto :
[sub.example.com...]
i see 404 page because the browser redircet to :
http://www.example.com/sub--robots.txt

[jdMorgan]

It doesn't matter that it did not work; The point of testing was to find out what the problem is.

Obviously the value for %1 is blank, so now the question is, why is %1 blank?

Please re-post that whole rule (RewriteRule and all RewriteConds) changing only the domain name to "example".

Thanks,
Jim

[egyption dev]

thanks Jim :

===================

RewriteCond %{HTTP_HOST} ^example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com:[0-9]+$ [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

RewriteCond %{HTTP_HOST} ^www\.([^.]+)\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.www\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com:[0-9]+ [NC]
RewriteRule (.*) [%1.example.com...] [R=301,L]

RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com
RewriteCond $1 ^robots\.txt$ [OR]
RewriteCond $1 ^sitemap\.xml$ [OR]
RewriteCond $1 ^google[0-9a-f]{16}\.html$ [OR]
RewriteCond $1 ^y_key_[0-9a-z]{16}\.html$ [OR]
RewriteCond $1 ^LiveSearchSiteAuth\.xml$
RewriteRule (.*) /home/user/public_html/sub-%1-$1 [L]

RewriteCond $1 !^sub-
RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com
RewriteRule (.*) /home/user/public_html/category.php?slug=%1 [L]

===================

I'm sorry for tired you (weak language :))

[jdMorgan]

Your English is fine, but that is not the rule you tested... I'd like to see the one with the [R=302,L] that caused the 404 and the redirect to www.example.com/sub--robots.txt

Jim

[egyption dev]

thanks for help

===================

RewriteCond %{HTTP_HOST} ^example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^www\.example\.com:[0-9]+$ [NC]
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

RewriteCond %{HTTP_HOST} ^www\.([^.]+)\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.www\.example\.com [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com\. [NC,OR]
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com:[0-9]+ [NC]
RewriteRule (.*) [%1.example.com...] [R=301,L]

RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com
RewriteCond $1 ^robots\.txt$ [OR]
RewriteCond $1 ^sitemap\.xml$ [OR]
RewriteCond $1 ^google[0-9a-f]{16}\.html$ [OR]
RewriteCond $1 ^y_key_[0-9a-z]{16}\.html$ [OR]
RewriteCond $1 ^LiveSearchSiteAuth\.xml$
RewriteRule (.*) http://www.example.com/sub-%1-$1 [R=302,L]

RewriteCond $1 !^sub-
RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^([^.]+)\.example\.com
RewriteRule (.*) /home/user/public_html/category.php?slug=%1 [L]

===================

[jdMorgan]

OK, this is the only rule we need to worry about right now:

RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteCond %{HTTP_HOST} ^[b]([^.]+)[/b]\.example\.com
RewriteCond $1 ^robots\.txt$ [OR]
RewriteCond $1 ^sitemap\.xml$ [OR]
RewriteCond $1 ^google[0-9a-f]{16}\.html$ [OR]
RewriteCond $1 ^y_key_[0-9a-f]{16}\.html$ [OR]
RewriteCond $1 ^LiveSearchSiteAuth\.xml$
RewriteRule (.*) http://www.example.com/sub[b]-%1-[/b]$1 [R=302,L]

The value matching the sub-pattern in parentheses in the second RewriteCond should be getting put into the RewriteRule substitution URL (I made these two elements bold). However, this back-reference function is not working for some reason; The value for %1 appears to be blank, because you report seeing a redirect to "/sub--robots.txt".

However, the back-reference to $1 (value="robots.txt") *did* work when you tested it, and you did get a redirect as expected, but with the subdomain-name value missing.

There is no reason for this rule to fail in that way, and the only thing I can think of is some sort of character-encoding problem. You might want to very carefully re-type both lines into your code, and be sure to use a simple plain-text editor such as Windows NotePad to do this.

If it is not a character-encoding problem, I don't know what the problem could be... :(

Jim

[egyption dev]

ok , and thanks

i want to create one robots file for all subdomains
can i do that ?

thanks again; :)

[jdMorgan]

Yes, duplicate the code above, but testing only for robots.txt URL-path requests. The change the substitution path to remove the subdomain from sub-<subdomain>-robots.txt, making it only sub-robots.txt.

Jim

[g1smd]

I don't understand the logic as to how the %1 value is propagated to where it is used, some 6 lines lower. I thought any re-use applied only to the very next line of code. I do see $1 in the intervening five lines though.

[jdMorgan]

You can re-use (de-reference) back-references as often as you like. $1 and %1 persist independently, and replacing $1 will not affect %1, or vice-versa; they are separate variables.

Side note: One reason that RewriteConds are not processed until *after* the RewriteRule pattern matches is so that RewriteConds can use $1-$9 variables created in the RewriteRule. But since the RewriteRule substitution isn't invoked until after the RewriteCond patterns also match, the RewriteRule can make use of the %1-%9 back-references created by the last-matched RewriteCond.

Jim