Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- Bing ignoring robots.txt?


revrob - 7:20 am on Sep 29, 2012 (gmt 0)


Thanks for the helpful reply. I understand the second half of your reply about the separate Rules not being necessary and will implement your advice about [OR]

Presumably this is what you had in mind?

RewriteCond %{REMOTE_ADDR} ^157\.(5[4-9]|60)\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$ [OR]
RewriteCond %{REMOTE_ADDR} ^131\.253\.(2[1-9]|3[0-9]|4[0-7])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$ [OR]
RewriteCond %{REMOTE_ADDR} ^65\.52\.([0-9]|[1-4][0-9]|5[0-5])\.([0-9]|[1-9][0-9]|1([0-9][0-9])|2([0-4][0-9]|5[0-5]))$
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* http://www.example.com/robots.txt [L]

I don't understand the first bit of your reply and what you mean by "leave off the protocol-plus-domain part of the target". Could you translate that please with an example? And clarify what the difference is between a Rewrite and a temporary redirect, please especially in terms of what the visiting bot receives by way of a response.

As for the final part "that's for another forum ;-)" - I think I got that particular bit of code FROM another WW forum ;-) - where should I go to chat about that please? This WW site is where I have learnt everything (so far very little) I know about .htaccess and the help is much appreciated.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4441555.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com