TelegramBot

[keyplyr]

UA: TelegramBot (like TwitterBot)
Protocol: HTTP/1.1
Robots.txt: No
Host: Telegram_Messenger_Network
149.154.164.0 - 149.154.167.255
149.154.164.0/22

Telegram is a messaging app that uses "bots" connected to accounts.

Bots are simply Telegram accounts operated by software – not people – and they'll often have AI features. They can do anything – teach, play, search, broadcast, remind, connect, integrate with other services, or even pass commands to the Internet of Things

source: telegram.org/blog/bot-revolution

[lucy24]

Lightbulb moment: By putting the element “like TwitterBot” in the UA string, they can follow robots.txt directives that are intended for the bona fide TwitterBot, all while adhering to the letter of the robots.txt law.

[keyplyr]

Ya think that's why they (and others) do that?

I thought they do it to make themselves look friendly... "if you like Twitterbot, we're just like them so please allow us as well."

Actually, these bots (as described above) are nothing like Twitterbot.

[lucy24]

The real Twitterbot is anomalous in that it asks for and appears to obey robots.txt. I don't suppose these guys do, but they can always fall back on “But you said the Twitterbot is allowed to be here!”

:: wandering off to add ^ opening anchor to legitimate Twitterbot's whitelisting ::

[keyplyr]

I filter Twitterbot (actually "twitter") by authorized range, so anyone else using Twitterbot in the UA string would get blocked... at least at first until I look under the hood.

[lucy24]

Fakers don't generally go to the trouble of matching the real thing's headers, either.