Facebot / Twitterbot hit from AS6730 (Sunrise Communications AG)

[SumGuy]

I see in my logs today a hit from 178.38.119.X (what-ever.adslplus.ch). On the HTML side, a sequence of 5 file requests, the first one for an HTML file using this UA:

Mozilla/5.0 (iPhone; CPU iPhone OS 14_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML like Gecko) Version/14.0.2 Mobile/15E148 Safari/604.1

6 seconds later, all during the same second, 4 file requests, the first being the same HTML file, the 3'rd being favicon.ico, the last two for an apple-touch-icon png. These 4 requests had this UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

None of these have referrer. No previous history of any hits from 178.38/16

The two requests for the HTML file were re-directed (301) to my HTTPS server, where they requested the file (twice) using the same two UA's given above. No other files were requested that normally should be to fully render the page.

One other file (a different apple icon png) was requested on the HTTPS side, but using this UA:

MobileSafari/604.1 CFNetwork/1209 Darwin/20.2.0

The fact that no other files referenced in the HTML file were requested that normally would be by a human browser to render the page is to me a dead giveaway that a bot was behind this hit, but yet the requesting IP seems to come from a residential ISP (I have had hits from Sunrise IP's before, but not from this /16, and have attributed them to be organic/human).

If it matters in the context of this event, I do block Fecebook IP's and periodically see their attempts to retrieve files.

Is there indeed a combined, legit Facebot / Twitterbot? Does Fecebook have such a business affiliation with Twitter?

If this is bot activity (FB/TW or someone else) then Sunrise is renting IP's to them?

Or is this "in-app" bot behavior - bots hitting my site through someone's iphone?

[lucy24]

Someone hereabouts once explained “Facebot Twitterbot”. Possibly more than once, because the explanation just won’t stick. It is associated with a human, and isn't directly either FB or Twitter.

:: shuffling papers ::

Oh, here [webmasterworld.com]. See post from iamlost, halfway down the thread. It's associated with the iOS iMessages app, whatever the heck that is. (Hm. I just got a new iPad, so it's possible I even have iMessages and don't know it. So far all I've done is buy a fresh copy of sudoku.)

Darwin is something involving images, again on mobiles.

[dstiles]

I had a bad hit from sunrise yesterday. I put it down to a botnet hit from a virused machine. I get a lot of those (relatively speaking), but maybe not.

Facebook itself, I think, can no longer claim to just access links posted on the site. The bot now acts more like an ordinary SE bot. I've had several hits from it on sites that no one in a right frame of mind would ever post to facebook (or twitter).

[lucy24]

The bot now acts more like an ordinary SE bot.

Including, interestingly enough, asking for and apparently complying with robots.txt. I think it was a couple months back that several people reported this unexpected behavior. But most of the time I just see the ordinary FB.

Now, personally I think it would be more appropriate for FB to use an entirely different UA for their general crawling than for their normal activities, so site administrators can handle them differently if they so choose. But these are the people who once decided there was no need to send any UA at all. It took them about a year and a half to figure out that the only effect is getting your visits blocked from many sites.

[SumGuy]

Got an interesting email today. Over the past 48 hours I've been getting a ton of "I know you watch #*$! I've got photos of you taken from your camera I've hacked your email see how I'm sending you an email from your own account now send me bitcoin and I'll go away" spams. I'm sure I've blocked (by IP) many hundreds of these on my mail server (seeing a big spike in rejected IP's lately) but some are getting through. One of the ones that came in today came from the same /16 Sunrise IP subnet as the facebot / twitterbot hit I describe in my first post. Sunrise doesn't seem to have a lot of IP's (800k?) so this is some coincidence.

[dstiles]

Probably not sunrise's fault directly, any more than any broadband provider. Unfortunates and idiots get viruses. Some take a few hours to notice and eradicate, others take days or weeks, some never do. Sadly, in some cases it's hosting and broadband providers who get hacked; this may or may not be true in this case. As far as I recall sunrise is Swiss, so one expects a slightly more responsible action.

The sadness is that ISPs are not (normally) allowed to block infected customers and hosting providers don't usually bother.

[SumGuy]

> The sadness is that ISPs are not (normally) allowed to block infected customers

Blocking port-25 outbound on dynamic / residential IP space is beyond common practice today, at least in western countries, or at the very least in US/Canada. Based on past spam, I apparently was already blocking about 2/3 of Sunrise IP space (port 25 that is) and it's now 100%. A legit sunrise customer will most likely have a gmail account (or equivalent) and use it if they did ever want to send us an email.