Brett_Tabke - 9:20 pm on Apr 22, 2000 (gmt 0)

This is rather bizzare, but I've setting here trying to get Apache to execute an ssi on a TXT file (very tricky, but doable). Once that is done, I can run a logger from the robots.txt and monitor pulls.

The only other way to monitor robots.txt is via your server logs. If you don't have a robots.txt then look in the site error log for "file not found".

Hitbox can not track a spider. Webtrends can of course since it uses your server log files, but hit box is a graphic counter only (records about 75% of your hits).