I've been having bot troubles. A site of mine has been exhaustively downloaded by a range of bots, and this is quite annoying because I don't know why (email search, content scrape?).
I'm working on a range of counter measures, and one that I came up with is based on the fact that most of the bots requested gzipped data. (No doubt so they can save their own bandwidth whilst stealing my content faster.
I have a php script that streams a multi-gigabyte decompression bomb to anyone who hits it. What is the best way to link the file, such that it won't get hit by innocent bots or human users?