Forum Moderators: open

Message Too Old, No Replies

The Knowledge AI

The Knowledge AI

         

WebOpz

12:03 pm on Apr 21, 2021 (gmt 0)

5+ Year Member Top Contributors Of The Month



Been seeing a bot that doesn't attempting to scrape my site. It doesn't include contact info or seem to listen to my listing it in my robots.txt. Anybody know who this is?

not2easy

12:15 pm on Apr 21, 2021 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



A search here for the UA turns up several mentions, some in depth info:
2018: [webmasterworld.com...]
2020: [webmasterworld.com...]

And about half way down the 2020 edition [webmasterworld.com] of lucy24's At Home With the Robots:
The Knowledge AI

IP: 66.160.140, 64.62.252
UA: The Knowledge AI

Distinguishing feature: Unable to do HTTPS. Over the years, as my sites move to HTTPS, it has picked up an increasing number of redirects, to the point where it can now retrieve no content except robots.txt. This does not, ahem, make them seem especially knowledgeable.


lucy24

4:03 pm on Apr 21, 2021 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



As far as I know*, the inability to do HTTPS remains. For robots.txt, use the exact full UA name. It's been a few years, but I seem to remember they didn't recognize a partial like "Knowledge" alone; I had to spell out the whole thing. I found them to be compliant once this was sorted out.

Since my standards are not very high, I've never bothered to find out who they are. If they comply with robots.txt and don't come from {country}, that is generally all I ask.

* “As far as I know” because I am temporarily without access to archived logs, but they do show up periodically in redirects.

WebOpz

2:30 pm on Apr 23, 2021 (gmt 0)

5+ Year Member Top Contributors Of The Month



Thank you both. =) LMAO at the description: " has picked up an increasing number of redirects, to the point where it can now retrieve no content except robots.txt. This does not, ahem, make them seem especially knowledgeable."