Forum Moderators: open
Should the fact it does not read robots.txt make it a bad bot that I exclude?
Or does the fact that the maximum requests it made in a 1 minute period were 22 with most of them at around 6 or 7 a minute make it a good bot even though it took stuff from folders that are excluded in robots.txt for all bots?
I'm trying to decide whether to ban it or not. Thanks in advance for your comments.