Forum Moderators: goodroi
The info page says it's a distributed crawler, so just like my policy for the cronic robots.txt violater Grub, I banned the user agent and the entire IP block associated with the offending IP.
Just out of interest, can you post your robots.txt?
My robots.txt validates and has been in use for a while if that's what you're wondering about. If you have another question about it, just let me know.
Webmasterworld categorizes my username as a new user, but I'm not actually a new webmaster.
This thread seems to be straying from its purpose which was to give a heads-up about a rougue bot. In this case, the boitho.com bot specifically fetched spider trap URLs that were (and have always been) disallowed in robots.txt. The bot did not request any other files on the site.
I am one of the people behind Boitho. The Boitho robot does follow the robot exclusion protocol, and should not crawl pages that are excluded.
Can you please send me the urls that what crawled, and tell me how old the robots.txt file is to mail address “runarb ( at ) boitho.com”? Sow I can look inn my logs to see what did go wrong.
Information about the boitho robot is available here: [boitho.com...]
Regards
Runar Buvik
You say that your bot is supposed to obey robots.txt. Is that hardcoded or a user option? If it's only an option, then the first malicious user that ignores robots.txt ends up getting your bot banned as in this case. If it's hardcoded but buggy, then of course the same result.
Both Grub and MJ12bot have been banned from all sites that I administer because of robots.txt violations. Now your bot has been added to the list. Maybe not all webmasters will be as quick to ban misbehaving bots as I am, but I've seen too much abuse on my sites to grant leniency and I certainly don't have time to hand-hold every bot writer who thinks they might have the next big search engine.
While it's too late for grub, boitho, and MJ12bot as far as my servers are concerned, the best suggestion I have for anyone else attempting to write a legitimate bot is to hardcode it to respect robots.txt and test it thoroughly against the spec before you release it. If you allow a user of your bot to override that or if you release your bot before you've corrected any robots.txt-related bugs, then you run the risk of having your bot summarily banned by a large number of webmasters and as a result, rendered useless.
EDIT: I forgot to mention the search was done from the bot page, not the main page.
Both Grub and MJ12bot have been banned from all sites that I administer because of robots.txt violations.
Now that you mentioned my bot I have to respond and ask to provide robots.txt that was supposedly violated by the bot.
the best suggestion I have for anyone else attempting to write a legitimate bot is to hardcode it to respect robots.txt and test it thoroughly against the spec before you release it.
Its hard coded and its not optional: users can't turn it off. The implementation is very robust and I had only half a dozen reports (after ~ 500 mln URLs crawled) about supposed violation of robots.txt of which only one was correct (that bug was fixed same day).
If you can't back your words with robots.txt (post it here since its the relevant forum) then it would be good manner not to accuse others of breaking robots.txt spec.
I certainly don't have time to hand-hold every bot writer who thinks they might have the next big search engine.
You sure have time to post on the subject -- all you need to do is to provide your current robots.txt or just sticky with URL. If you refuse to do so little to set the record straight, then I have no choice but to consider your allegations false and kindly ask to stop spreading incorrect information that you can't back up.
Now that you mentioned my bot I have to respond and ask to provide robots.txt that was supposedly violated by the bot.
I've already responded to that inquiry above.
Its hard coded and its not optional: users can't turn it off. The implementation is very robust and I had only half a dozen reports (after ~ 500 mln URLs crawled) about supposed violation of robots.txt of which only one was correct (that bug was fixed same day).
That sounds like you may have good intentions, but my logs show a violation and that's what I go by. And you just admitted that you have violated robots.txt on at least one occasion that was reported to you. I wonder how many webmasters just banned you outright like I did without filing a bug report or commenting.
If you can't back your words with robots.txt (post it here since its the relevant forum) then it would be good manner not to accuse others of breaking robots.txt spec.
What you regard as good or bad manner is not my concern. I've already responded to your robots.txt inquiry above and offered to answer any syntax questions.
You sure have time to post on the subject
Posting on the subject is to benefit others. I have no interest in helping you debug your bot even though I have offered to answer syntax questions.
all you need to do is to provide your current robots.txt or just sticky with URL.
Think about what you're asking. You're asking me to supply personally-identifiable information to an entity that has left evidence of malicious behavior on a site I administer. No thank you.
If you refuse to do so little to set the record straight, then I have no choice but to consider your allegations false
That's your prerogative. Personally I would not be so quick to dismiss a report of an error with my software, but everyone has their own policies. Of course, it's certainly possible that you may have corrected whatever bug caused your bot to violate my robots.txt, but so far I haven't seen any reason to lift the ban and your demeanor certainly does not help.
and kindly ask to stop spreading incorrect information that you can't back up.
The information is correct, I choose not to provide personally-identifiable information and I will post as I see fit.
