IP: 144.76.68.abc
UA: serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)
robots.txt: yes, compliant
Most robots walking in off the street would be summarily blocked, but this one had good enough headers that it waltzed right in--and proceeded to demonstrate robots.txt compliance by carefully avoiding a roboted-out directory that is linked from all pages. Goood robot.
Requests come in clumps, anywhere from a few seconds to several minutes apart, except that the root was requested immediately after robots.txt. On my personal site, 19 pages took a bit over an hour. (This is the only site they visited, so they probably really are following-up someone else's links.)
Barring information to the contrary, I’m going to put them in the “You scratch my back, I’ll scratch yours category”. For me this means: I do not personally use the service suchandsuch robot provides. But other humans do, and the service is improved by being able to crawl freely. Conversely, I use services that other humans may not use, but which again are more effective if they’re allowed to crawl. (Only consider how exasperating it is when you have to check a link manually because the w3 link checker wasn’t allowed in.)
