Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Is it illegal to ignore a robots.txt?
DaveAtIFG - 6:44 am on Feb 3, 2004 (gmt 0)
robots.txt is a convention, it's not even an adopted "standard" by any of the bodies that establish standards. See [searchengineworld.com...] and note these points:
It is not an official standard backed by a standards body, or owned by any commercial organisation.
It is not enforced by anybody, and there no guarantee that all current and future robots will use it.
Consider it a common facility the majority of robot authors offer the WWW community to protect WWW server against unwanted accesses by their robots.
You could sue Google or anyone else operating a spider that ignored robots.txt. It's very unlikely that you would prevail. If you did, it's doubtful that damages would amount to more than a few dollars for the bandwidth consumed by the spider.
Thread source:: http://www.webmasterworld.com/robots_txt/265.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com