tschild - 10:05 am on Oct 30, 2003 (gmt 0)

Another option would be to specify disallowed directories with a partial path which is only long enough to not include allowed directories.

Say you have the following files and directories:

/index.html
/my_sekrit_files/
/pages/
/sitemap.html
/stuff_that_i_dont_want_to_be_spidered/

then you only need to specify

User-agent: *
Disallow: /m
Disallow: /st

to disallow those two directories, and leave snoopers in the dark about the names of the excluded directories.