Apple applebot increasing activity

Increasing hits to interior web pages

[SumGuy]

Here is the year followed by the number of interior page-hits I've seen from the Applebot:

2015 5
2016 15
2017 41
2018 58
2019 27 (extrapolated for the year -> 131)

I'm counting hits from 17.anything with applebot user-agent. I'm not counting hits to my default / landing page or to favicon.ico or apple-touch-whatever.png files or robots.txt. By google/bing standards the number of applebot hits is very very low. But there is a trend here.

[lucy24]

Huh. I wouldn't have thought of constraining searches to interior pages, but now that you mention it, it is a useful metric. (What’s equally interesting is the proportions: interior pages as a percentage of total visits vary from a low of 32% to a high of 86% over the 4½ years I looked at. This year it’s 44%; last year it was 74%.)

:: detour for my own checks, run in the background while I proofread Rambles beyond Railways in foreground ::

Gosh, SumGuy, I think your site is smaller than mine--something I would have thought to be mathematically impossible. Unless your numbers stand for thousands or millions, ahem. Mine, treating ytd as 1/5 of the year to make it easier to multiply:

2015 138 (/160)
2016 151 (/278)
2017 197 (/608)
2018 5212 (/6976)
2019 475 (/1067) projecting to 2375 (/5335)

So for me it was 2018 that went through the ceiling.

:: closer look ::

Let’s look at this again in six months. Last year, the busiest Applebot months by far on my site were June-July-August; in fact the whole first quarter of 2018 had only about half as many visits as this year to date (call it 5/6 of the quarter).

:: double-take ::

Last year, almost 3/4 of ALL Applebot interior-page requests were for a single page. wtf? This is an ebook (public-domain text), so it’s not like the page content ever changes.

Has anyone ever figured out what, exactly, the Applebot does?

[SumGuy]

My site doesn't have too many pages. The interior pages that applebot is hitting are linked directly off my landing / default page. Other types of content (like PDF files) are a few levels down from the landing page and so far not being requested. I have seen a few hits from 17.what-ever that look like legit human hits (browser is not applebot and all page files like jpgs are being requested) - I don't know if sometimes Apple's bots impersonate human browsers... ?

[lucy24]

I don't know if sometimes Apple's bots impersonate human browsers... ?

If they do, they are better at it than microsoft’s bots ;) because no two human(oid) visits are exactly alike: different browser versions, slightly different OS--though always recent, of course. A lot of Safari, as you might expect, but also some Chrome.

Then again, in all of last year I find a whopping total of eight humanoid visits--one in 2016, none in 2017, a few earlier--seven of them on the same two dates in November. All three dates in 2018 were the same day of the week but I think we’d better call that coincidence.

a few levels down from the landing page and so far not being requested

I'm seeing almost the opposite. In particular, that excessively-visited interior page I mentioned above has definitely never been linked from the root. In fact, very few of those thousands of requests are for pages that are directly linked from the root. Not absolutely zero, which would raise suspicion, but comparatively rare.