What on earth is this bot? At the moment I'm getting like a hit on my website every two seconds from this. IP at the moment is 66.160.140.184 and user agent is "The Knowledge AI". Any clues? Google didn't really know anything special.
not2easy
4:44 pm on Jul 18, 2020 (gmt 0)
Depends on where Google is looking. If you use Google to search WebmasterWorld, it turns up lots of information - I got this: [google.com...] ;)
For some specifics, even the same bot host is noted here in 2018: [webmasterworld.com...]
The bot was described more fully in this year's At Home With the Robots article: [webmasterworld.com...]
No reason to permit imho.
lucy24
8:01 pm on Jul 18, 2020 (gmt 0)
I have no idea what it does, but I tend to be lenient with robots, so I authorized it long ago. This, however, has become moot, as it turns out to be one of the rare law-abiding robots that is only able to do HTTP. This means that for the last eight-nine months it has received nothing but 301s (except robots.txt, which I don’t canonicalize).
In addition to 66.160.140.various, I also see it from 64.62.254.various.
tangor
2:44 pm on Jul 19, 2020 (gmt 0)
It has always been well behaved for me, just asking for robots.txt and nothing else.
Everyone gets a bite of robots.txt on my sites, even if I determine they are not following the rules specified ... makes it easier to nuke their bad ways (and UAs which are NOT always the same).
