um-LN

[keyplyr]

UA: "Mozilla/5.0 (compatible; um-LN/1.0; mailto: [e-mail]
Protocol: HTTP/1.1
Robots.txt: No
Host: hetzner.de
88.198.0.0 - 88.198.255.255
88.198.0.0/16

Media monitoring for marketing products

[lucy24]

Oh, cripes, ubermetrics again?

:: detour to files ::

I’ve got them flagged as “blocked regardless”. Variably uMBot-FC and um-FC. I would say

Robots.txt: Yes but ignores

although I now see I've never bothered to do the final test: Do they require a Disallow: block of their own?

What's the difference between LN and FC? Neither one is a country ... and that's all I know.

[keyplyr]

Do they require a Disallow: block of their own?

"disallow?" If they're not supporting robots.txt, I doubt disallowing them will do the trick.

As far as blocking, I would expect you to be controlling everything coming from hetzner.de. While I whitelist a few agents from hetzner.de, ubermetrics-technologies.com isn't one of them.

What's the difference between LN and FC?

They both get an "um" from me.

[lucy24]

If they're not supporting robots.txt, I doubt disallowing them will do the trick.

My ordinary robots.txt goes like this:

User-Agent: you
User-Agent: and you
User-Agent: and you too
Disallow: /

A few robots do not understand this structure, and require their own, blank-line-delimited sector:

User-Agent: just you and nobody else
Disallow: /

Rare, but it does happen.

[keyplyr]

User-Agent: you
User-Agent: and you
User-Agent: and you too
Disallow: /

That is about the *least* supported format. Most require separate directive for each UA.

Here again we're reminded that making the robots.txt a standard, while a worthwhile endeavor, failed. Most UAs do not support it and those that do have different requirements. It is fast becoming archaic.

[lucy24]

That is about the *least* supported format.

I've found it works at least ¾ of the time, so I always try it first.

From beginning to end, the one absolute standard is

User-Agent: this-means-you
Disallow: /

If they don't appear to understand this, there was no point in asking in the first place. And sometimes asking-and-ignoring is worse than not asking at all.

[keyplyr]

I keep a current robots.txt also, and am pleased when a bot supports it so I don't have to resort to other methods - however, I just don't base my opinion of an agent if they don't ask or support it.

[lucy24]

I just took a closer look. Turns out I do have a separate this-means-you block ... but only for the former UA, um-FC. I have now changed it to “ubermetrics” which seems to be the recurring element. We Shall See.