How to call different robots and sitemap on same server?

[AnnRick]

Hi,

I have 2 domains and currently both domains using same robots.txt (hosted on same server). I need different robots.txt and sitemap.xml. Below is the code I added in .htaccess but not working..

## robots.txt for multi store ##
RewriteCond %{HTTP_HOST} ^.*?domain\.com$ [NC]
RewriteRule ^robots\.txt$ robots/domain.com.txt

RewriteCond %{HTTP_HOST} ^.*?domain\.co\.uk$ [NC]
RewriteRule ^robots\.txt$ robots/domain.co.uk.txt

What is wrong in it? Please help to resolve.

Thanks in advance.

[penders]

Your problem is probably the relative path substitution in the RewriteRule. See this thread in the Apache forum:
[webmasterworld.com...]

Also, I assume you have enabled the rewrite engine?

RewriteEngine On 
RewriteCond %{HTTP_HOST} domain\.com$ 
RewriteRule ^robots\.txt$ /robots/domain.com.txt [L] 
 
RewriteCond %{HTTP_HOST} domain\.co\.uk$ 
RewriteRule ^robots\.txt$ /robots/domain.co.uk.txt [L]

Checking for the HTTP_HOST could perhaps be tidied some more. What are the possible values? Do you need to check for both www and none-www versions? Presumably you are already redirecting www/none-www earlier in your script?

[AnnRick]

Yes, rewrite engine is on. My website is built on magento and I want to achieve it for multi store with different domains.

I tried with below code but still not working.
RewriteCond %{HTTP_HOST} domain\.co\.uk$
RewriteRule ^robots\.txt$ /robots/domain.co.uk.txt [L]

HTTP_HOST is 'domain.com'. I am checking robots.txt with www version. There is no redirection rule set for www/non-www in .htaccess.

[lucy24]

You can't redirect a request for robots.txt, because major search engines won't follow the redirect. If you're not prepared to mess about with proxies-- which frankly seems more trouble than it's worth for a robots.txt request-- you need a separate physical robots.txt file in each domain's directory. And then it's a lot easier if you just leave the robots.txt in the root directory, where the server will find it by default with no rewriting.

RewriteRule ^robots\.txt$ /robots/domain.co.uk.txt

This rule says: If there is a request for robots.txt in the specified domain, quietly serve up the file living in the same domain's top-level /robots/ directory. Is that what the rule is intended to do?

But, again, why not just stick with defaults and save the server the extra work?

[penders]

You can't redirect a request for robots.txt, ...

Although these are internal rewrites.

I assume the OP has two domains pointing to the same webspace, one or both are "parked" domains on a single hosting account. (A single code base serving, what appears to the user as, two separate sites.)

[lucy24]

Although these are internal rewrites.

Yes, I was specifically referring to the word "redirect" in the second post. When someone is asking about internal rewrites, it makes me uneasy even to see the word "redirect" ;)

Looking more closely ...

There is no redirection rule set for www/non-www in .htaccess.

That shouldn't have anything to do with the problem, though I now realize it's an exception to the "no robots.txt redirect" principle. If a request for
example.com/robots.txt
is redirected to
www.example.com/robots.txt
they will by-and-by ask for robots.txt at the right place. (Although sometimes only after asking for example.com/ and getting redirected there too.) But this happens before the internal rewrites. (And, ahem, it should happen. Add the rule if it doesn't exist.)

Incidentally, this locution

^.*?

(with no capture) is never necessary. Just leave off the opening anchor and start with the part you need to match.

what appears to the user as two separate sites

I hope it doesn't appear that way to search engines too, or we're talking Duplicate Content all over the map. Quadruplicate, if there's no with/without www. redirect.

[penders]

Your problem is probably the relative path substitution in the RewriteRule. See this thread in the Apache forum:

Actually, this might not be the problem - although relative paths are never a good idea here. (I should have followed up on the other thread first!) The issue with relative paths (that I was referring to) appears to be more prevalent with redirects, not rewrites (as we are doing here)!

There is no redirection rule set for www/non-www in .htaccess.

That shouldn't have anything to do with the problem...

Agreed. My only query with regard to www/non-www, was that if you knew the user would only be requesting say "www.example.com/robots.txt", then you would only need to check for exactly "www.example.com", not both.

what appears to the user as two separate sites

I would expect/hope this to be two entirely separate sites... different content, images, URLs, ...design? The Magento CMS would decide what to serve based on the HOST? (But, yes, otherwise it's duplicate central!)

... but still not working.

Can you identify what is happening? Is it doing anything? Do you have other directives in your .htaccess file that are perhaps conflicting? Do you have nested .htaccess files?