Brett_Tabke - 5:25 pm on Sep 25, 1999 (gmt 0)
About 6 months ago, they hit one of our
sites completely. It was on the old isp and I didn't have access there to robots.txt. So, I whipped up a perl script to "toy" with them (letting stuff timeout, sending them junk - that sort of them). They came back a week later and hit everything in site with a slightly different user agent (including cgi's). I whipped up another script that was a dynamically created loop off .htm files (create a file, pull a file, create a file...). It sat there pulling that page loop for a full 20 hours. They've kept coming back and back. Evidently they think of me as a test site now. They routinely walk down the tree of all those dynamic graphics pages and most are form style posted urls once you get two pages deep.