7_Driver - 1:59 am on Dec 30, 2011 (gmt 0)

>There is no law against it, after all.

That remains to be seen...

I think the computer missuse act in the UK prohibits unauthorised access. You could argue that anything on a web server is fair game - but if the webmaster has correctly implemented industry standard files and tags to keep robots out - his intention is pretty clear. If the robots choose to ignore his instructions and access the files anyway, then perhaps they fall foul of that act.

I'm not a lawyer - but I reckon it might be arguable at least. Just because you CAN get something off a computer, doesn't mean you're allowed to.