Receptional_Andy - 9:14 pm on May 14, 2008 (gmt 0)

From my experience, historically, you had to link to robots.txt to get it indexed. These days, search engines seem to want to index absolutely any valid URL. This type of spidering is stretching the limits of robots exclusion.

If you don't want spiders to index certain URLs, your only recourse is to block them, preferably at a point in the network chain as close to the request as possible.