Page is a not externally linkable
- Microsoft
-- Bing Search Engine News
---- Bing ignoring robots.txt?


revrob - 11:05 am on Sep 29, 2012 (gmt 0)


So to make a rewrite - that the bot can't avoid:
would this do?

RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* /robots.txt [L]

And the redirect: - that the bot can see and then just go away
- is this correct?
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* http://www.example.com/robots.txt [R=301][L]

Does the choice of which one to use actually make a difference to frequency of unwanted visits from a badly behaved bot like bingbot? Or does it just speed up their visit and get rid of them quicker?

Once again many thanks for your patience.


Thread source:: http://www.webmasterworld.com/msn_microsoft_search/4441555.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com