I got an idea... not sure it will work. If I made txt-files run through PHP with an AddType in htaccess i could have a script in my robots.txt that recorded the IP. Then I would KNOW it was a robot without having to mess with robot-IP-lists.
3:59 pm on Jan 12, 2004 (gmt 0)
That may work but I've noticed a lot of spiders that don't look at robots.txt
6:28 pm on Jan 15, 2004 (gmt 0)
Also some browsers and internet accelerators and such look at robots.txt. And, robots don't always access robots.txt prior to accessing other files. Some robots only access robots.txt once a month or so, and continue to request files all month long.