Page is a not externally linkable
revrob - 11:05 am on Sep 29, 2012 (gmt 0)
So to make a rewrite - that the bot can't avoid:
would this do?
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* /robots.txt [L]
And the redirect: - that the bot can see and then just go away
- is this correct?
RewriteCond %{REQUEST_URI} !^/robots\.txt$
RewriteRule .* http://www.example.com/robots.txt [R=301][L]
Does the choice of which one to use actually make a difference to frequency of unwanted visits from a badly behaved bot like bingbot? Or does it just speed up their visit and get rid of them quicker?
Once again many thanks for your patience.