Inbound link analyzer, mentioned a few times in other forums. Says it respects robots.txt. Of course it would need to request robots.txt to do so.
lucy24
4:52 am on Dec 22, 2016 (gmt 0)
Assuming their UA is always truthful--and why the heck would someone claim to be the spbot if they're not?--they also hang out at 45.55.cc.dd 104.236.cc.dd 107.170.cc.dd 159.203.cc.dd and possibly other locations I haven't met yet. Expect them from all Digital Ocean properties, in other words. On each individual visit they pick an IP and stick with it for the duration.
I've been ignoring them since the beginning of October. (We have previously established that my standards are extremely lax for most purposes.) Since then they've visited six times. Raw logs say each visit was exactly 440 requests, from which I deduce that I have 438 robotable pages on this particular site. They always start with a redirected request for the root, which I suppose means they default to with-www.
Further investigation reveals that each time they visit my personal site--again, six visits in the last three months--there are 22 requests. This time, the redirect is for robots.txt. This would appear to mean that by default they ask for example.com/robots.txt without www, followed by www.example.com/ (root) with www. Weird.
keyplyr
5:03 am on Dec 22, 2016 (gmt 0)
Nice log-wrangling :)
Expect them from all Digital Ocean properties, in other words. On each individual visit they pick an IP and stick with it for the duration.
