Forum Moderators: goodroi
The info page says it's a distributed crawler, so just like my policy for the cronic robots.txt violater Grub, I banned the user agent and the entire IP block associated with the offending IP.
Rejecting offers to supply more information...
So he rejected the robots.txt file you sent him?
Rejecting offers to supply more information is another easy way to dismiss reported bugs.
The information that matters is the robots.txt file and URLs to check that you refused to provide.
Its like saying that a browser is not rendering page correctly and then refusing to provide HTML to verify bug report.
So you finally supplied him with the only info that would matter
<Sigh> As has been covered already in this thread, there's no way to know if the information I offered would have mattered because the offer itself was rejected -- no information was ever examined.
the guy never supplied the robots.txt
Already covered multiple times in this thread.
instead he offered him his advise as consultant :P
Your sarcastic remark highlights a big part of what caused this thread to degenerate so quickly, i.e., assuming that a report is inaccurate or that a reporter is a rookie.
Already covered multiple times in this thread.
Its a rather safe guess that it would have taken you less time to re-type the whole of your robots.txt manually using small finger on the left hand than to post as much as you did in this thread.
So he rejected the robots.txt file you sent him?
Already covered multiple times in this thread. Since this is an old thread, I think some people might benefit from re-reading it before they post as most of the recent questions have already been covered.
...or that a reporter is a rookie
A typical rookie mistake is not providing the necessary data to fix a problem - ex: telling someone their spider is broken, then not providing their robots.txt and a sample URL.
Its a rather safe guess
More assumptions.
that it would have taken you less time to re-type the whole of your robots.txt manually using small finger on the left hand than to post as much as you did in this thread.
Okay, now you're just trolling. Supplying the robots.txt has already been covered multiple times in this thread. Do you really want to spend another three pages saying the same things over and over again?
A typical rookie mistake is not providing the necessary data to fix a problem
I always ask for specific information that helps track bug, and asking for robots.txt is pretty much given, so much that I never came across with someone actually refusing to show it -- jazzman is the first, and hopefully last.
Its as if the guy does not want his problem fixed, or the problem did not exist in the first place. Note that he started this thread accusing boitho spider of violating robots.txt, but then quickly switched to bash my bot, in both cases not providing any details that can reasonably help investigate the problem.
A typical rookie mistake is not providing the necessary data to fix a problem - ex: telling someone their spider is broken, then not providing their robots.txt and a sample URL.
You're way behind -- try to keep up. That was covered way back at the beginning of the thread.
