Sorry that I probably don't have the format correct.
UA : DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot) Robots text : No Host : Hertzner.de 148.251.121.91 136.243.0.0 - 136.243.255.255 136.243.0.0/16
Martin Potter
8:19 pm on Mar 7, 2020 (gmt 0)
Correction -- He *did* check the robots.txt file! (For some reason I missed seeing it in the log.)
lucy24
11:30 pm on Mar 7, 2020 (gmt 0)
I see DomainStats every month or so from assorted IPs in 148.251 and 136.243. They have a fairly consistent pattern: robots.txt HEAD / (root) GET / (root) and then all authorized pages linked from the root.
Requests seem to be dynamic rather than following a shopping list, because links from my root change periodically and they always request whatever is currently linked. This, in turn, implies that their default request is without-www, because otherwise there would be a visible redirect. (They don't seem to be interested in my personal site, which is with-www, so I can't be sure.) So far their default for the root has been HTTP; the site has only been HTTPS a few months, so again I don’t know if this will eventually change.
Visits are quite leisurely; the latest one took almost an hour to make 17 requests. Or 19, I guess, with the redirect. They punctiliously requested robots.txt both ways. (I realize it is technically possible to serve different robots.txt depending on whether the request comes through as HTTP or HTTPS, but so far I haven't met a case of someone actually doing this. Can’t figure out if it’s like with-and-without www where the two could genuinely be different sites.)
Are German robots grammatically masculine? :: detour to Cassell’s Dictionary :: So they are. How adorable.
Martin Potter
3:52 pm on Mar 8, 2020 (gmt 0)
Thanks, Lucy. I had not seen them before for some reason; guess they just discovered my site. After a quick search, the only reference I found to them here was a posting from 2016 with no replies, so figured they had gone away.
Interesting -- der robot, rather than die or das robot. Us guys are always in bad company.
tangor
6:22 am on Mar 9, 2020 (gmt 0)
Us guys are always in bad company.
Really? I strongly resemble that remark, and don't you forget it!
Martin Potter
12:35 am on Mar 12, 2020 (gmt 0)
Ahem, noted! (Thanks, tangor. You remind me of an old friend of mine. He was a real character, and very knowledgeable, who could be counted on for very insightful remarks, though always disguised as jest.)
