chiyo - 5:41 am on Apr 30, 2003 (gmt 0)

just a reminder that robots.txt only SUGGESTS to a ROBOT that it shouldnt INDEX certain directories. Even if the robot respects robot.txt through its policies (there is nothing illegal about a robot not respecting robots.txt though it is not good manners and can end up with them being banned on a site by site basis), it does not have anything to do with even suggesting that it should not be "looked at" or downloaded, or used by a "non-robot", or used in other ways.

It makes sense to suggest to a robot that they should not index your css, cgi, admin or other directories and should be seen as helpful.

But Robot.txting out css directories is useless if you are only trying to "hide" CSS invisible text tricks from Search engines.

[edited by: chiyo at 5:57 am (utc) on April 30, 2003]