-- Search Engine Spider and User Agent Identification
GaryK - 2:29 pm on Jun 21, 2009 (gmt 0)
Seems to me the long-standing, oft' proffered claim that the bot obeys robots.txt is specious/misleading when the MJ12bot doesn't ask for it in the first place.
Every time I've added the isBanned property to these bots I get a sticky or an e-mail claiming it does too obey robots.txt and then one of the bots comes calling and sure enough it reads and respects robots.txt. So I'm left to wonder, if it can obey robots.txt, why doesn't it do so all the time.