Psst! If you put the UA in [ code ] markup, it prevents auto-linking:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
Now, here’s an oddity. Looking through recent logs, I find that since mid-December of 2024, Amazonbot has requested
nothing but robots.txt, where it is ...
...
Whoops! Thanks to that “Chrome” in the UA, it’s been getting a comprehensive Disallow. My bad; it’s been authorized (in htaccess) for a goodish while. But it does tell me that, from any IP, it appears to be fully robots.txt compliant.
I have now made a change, and will see how it behaves in future weeks. In particular, will it stay out of Disallowed directories?
I should note that the version of robots.txt that it has been getting is the minimalist
User-Agent: *
Disallow: /
It is possible that some robots respond better to this form than to a more complex robots.txt.
