Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- Bing ignoring robots.txt?


lucy24 - 2:33 am on Sep 29, 2012 (gmt 0)


RewriteRule .* http://www.example.com/robots.txt [L]

Hate to break it to you, but that isn't a Rewrite. It's a temporary redirect. To keep it as a rewrite, leave off the protocol-plus-domain part of the target.

You don't need all those separate Rules. If the IPs won't all fit on one line, make them into separate conditions separated by OR.

RewriteCond %{REQUEST_URI} !/robots\.txt$
RewriteCond %{REQUEST_URI} !^/robots\.txt$

The second line is contained within the first line, so it's superfluous.

But, er, that's for another forum ;)

I locked out Yahoo ages ago. Belt and suspenders: IP block and UA both.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4441555.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com