Forum Moderators: open
70.90.215.8z - - [26/Feb/2008:23:51:43 -0600] "GET /robots.txt HTTP/1.0" 200 4549 "-" "Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)"
read robots (which has included the UA for an eternity and grabbed a single page.
No images.
The Class D website is active, although without any active pages.
There are several builds. One is a 3rd party hosted free site-search utility that I use on one of my sites. This one does follow the robots.txt standard and there is no way to alter that.
Other releases of this bot are packaged as a back-end directory builders similar to FAST and are available for public/cooperate purchase. Since this version is a commercial product, I would imagine the modules can be customized. I block this version since it has the potential to be used as a scraper.