Forum Moderators: open
Received a 403 from my side, so I don't know what it does exactly.Even if it had received a merry 200 it's very unlikely that you would have been any closer to finding out what its purpose was. The overwhelming majority of robots--other than the ones looking for vulnerabilities, which may not ask for the root at all--just grab your front page and go on their way. Often the 403 itself gives them the information they need, like turnaround time, or the bare-bones "this site exists".
...are all those root requests in my logs coming from robots that do read the HTML and then go away dejectedly because they are intelligent enough to know that these aren't the droids they're looking for?You're personifying what's merely code, written to perform a specific task, nothing more. It doesn't make judgements or decisions. Just because we refer to them as bots, spiders, crawlers, etc, doesn't make them anything more than just a few lines of code ;-P
