Page is a not externally linkable
scareduck - 11:47 pm on Jun 24, 2003 (gmt 0)
This way, non-annoybots see themselves as being restricted (assuming they care, which they probably won't), and everyone else sees a blank robots.txt. Simple, no?
Of course, if you're using PHP, you could tie robots.txt to a robots.php file, and do something like
<?
if (preg_match('/Bozobot¦Unruly Crawler v1.0¦Etc/i', $USER_AGENT)) {
print "User-agent: $USER_AGENT.\n".
"Disallow: /\n";
}
?>