lucy24 - 4:52 am on Feb 28, 2013 (gmt 0)

if I only returned X-Robots-Tag: noindex , they could still be downloaded in a drive-by fashion

They're not mutually exclusive approaches, though. First step should be to slap the no-index header on all your pdfs and pull them out of g###s index. This by itself will cut way back on people going straight for the pdfs, because now you've only got the people who already know about them.

I think it's pretty closely analogous to hotlink protection. But since the pdfs are linked from the pages rather than displayed inline, you can set up alternative accesses for human users whose browsers don't send a referer. And meanwhile you can redirect referer-less requests to a page that says "I'm sorry but..."