The above is a product lets users select stuff to download from the web and a search turned up no mention of it here. The URL for the product is the same as the user agent name.
Should the fact it does not read robots.txt make it a bad bot that I exclude?
Or does the fact that the maximum requests it made in a 1 minute period were 22 with most of them at around 6 or 7 a minute make it a good bot even though it took stuff from folders that are excluded in robots.txt for all bots?
I'm trying to decide whether to ban it or not. Thanks in advance for your comments.
If you dont want it downloaded and taken then it would be bad. Most of these products dont have any real use other than pulling the website for either offline usage or to take the code out of it. I would ban it if it bothers you. If it doesnt dont worry about it.