Robots.txt: No
Actually, Yes--but you may not have noticed (or may have deliberately chosen to count it as a No) because it typically requests robots.txt a couple of days before a crawl, and then works off the same copy for quite a while. But eventually it does comply. It must, or I wouldn't have poked a (header-based) hole for it.
A more striking feature of this robot is its ongoing dimwittedness. It constantly asks for things like
/filename.html
when the true form is
/directory/filename.html
as if it can't understand relative links. It also habitually asks for pages at Site A that have never existed there, but only came into being after moving to Site B (to which most of Site A now redirects, file by file).
With all this, it's surprising that it
does seem to know how to pick its name out of a list, as in
User-Agent: something
User-Agent: SemrushBot
User-Agent: something-else
Disallow: blahblah
(Aside: To date, I've only met one robot that doesn't understand this construction, but does become compliant when given a "Disallow" block of its own.)
