7_Driver - 1:59 am on Dec 30, 2011 (gmt 0)
>There is no law against it, after all.
That remains to be seen...
I think the computer missuse act in the UK prohibits unauthorised access. You could argue that anything on a web server is fair game - but if the webmaster has correctly implemented industry standard files and tags to keep robots out - his intention is pretty clear. If the robots choose to ignore his instructions and access the files anyway, then perhaps they fall foul of that act.
I'm not a lawyer - but I reckon it might be arguable at least. Just because you CAN get something off a computer, doesn't mean you're allowed to.