iPhone, safari and bots

[dstiles]

Yesterday, a customer tried out his new web site using an iPhone. We were trying to discover why he could not access it but that is not the story.

I directed him to one of my web pages which reported the IP and User-Agent, with odd results. The User-Agent he reported back from there was most unlikely...
Phone (Mobile Mode)

So I looked at the log for that access and got three results for it, all at the same time to the nearest second.

apple-touch-icon-precomposed.png (not available so returned 404)
UA: User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

browserinfo.php (the required page)
UA: User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 15_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Mobile/15E148 DuckDuckGo/7 Safari/605.1.15

index.php (no idea why; not requested by customer)
UA: User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

These were the only hits, all to the same second. It seems as though the actual page hit (browserinfo.php) was accompanied by the icon request and this was followed by an automatic hit on the home page, which the customer was unaware of.

I can accept the middle UA as being Safari including DDG as its default SE and advertising that fact. If it were MY web browser I would be worried by the first and third UAs. The customer has no facebook or twitter account, access or activity so it's not an expected action. It appears from this that Safari is reporting back to twitter and facebook at least some information about web page visits. This does not happen from a desktop Mac, which has a UA of...

User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15

Since this is a development site with a private domain name, this is not acceptable. yes, I know a domain cannot be completely private but to sneek information in this way is unexpected.

[Dimitri]

Is it the same if the URL is typed directly into the browser's address bar?

I remember reading about a UA "Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0" being related to the iMessages crawler.

It's possible the Safari's bot (automatic requests like for the icon), is identifying itself, including social media bots, to limit risks to be blocked.

Was the request to index.php, directly to the .php file, or was it a request to / that your server routes to the index.php file? Was it a GET request? or just a HEAD one? It's possible that Safari is checking the home page, for all kind of reason, like a preview, or searching for og tags, etc... (which in my opinion, is not right, of-course).

Just random thoughts.

[dstiles]

I see no reason to identify itself with bots. It's a valid UA, why lie?

The request was specifically / which I now think was read (GET) to discover the location of the apple and fav icons. My concern was and still is, however, the multiple bot claims. Was it sending on the page info to the bot-owners?

[lucy24]

Try it out with a few other people's iPhones. Is it possible this individual user has a phone setting that automatically does the iMessages thing?

index.php is definitely odd. I took a quick look through my own logs because I see Facebot Twitterbot occasionally, but around 7/8 of requests are for various kinds of favicon. Never an index.php.

:: further detour to logged headers ::

Oh, oops. The reason all the non-icon requests are blocked is that I set it as a bad_agent and then forgot I'd done so. They can have the icon, should they desire it, but enough is enough.

[not2easy]

I agree with lucy24 because there are so many levels of permissions on various apps that if a person were to simply download an app and use it with default settings, there may be a number of unwanted things happening.

[dstiles]

Lucy - I have few customers now and don't know of any with an iPhone, but you may be right. Personally, I know almost nothing about such devices, not owning a smart phone myself.

I'm inclined to my previous comment now, that the home page was requested to see if there was a meta tag specifying the location of the apple icon.

not2easy - as I say, I have no real knowledge of such things but you could be right.