Msg#: 3461188 posted 9:02 am on Sep 26, 2007 (gmt 0)
MSN Live search is one of the only mainstream search engine that keeps getting caught up in my bot trap, which is, obviously, forbidden in my robots file. What's their problem? Why do they visit files they shouldn't be? Shouldn't they be focusing instead on evaluating real pages instead? It's not like they're sending any real traffic anyway... that's just another strike into tolerating their bot, but my patience has limit.
Msg#: 3461188 posted 3:20 pm on Oct 6, 2007 (gmt 0)
A useful technique for this situation is to detect known-good 'bot requests for your 'trap' URLs, and internally rewrite them to a minimal page containing a link to your home page and a <meta name="robots" content="noindex"> tag.
Yes, it's cloaking, but with no intent to deceive anyone.