Forum Moderators: open
63.229.140.41 - - [29/Jan/2001:19:08:23 +0100] "GET / HTTP/1.0" 200 6165 "-" "Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; ....../1.0 )"
The robot does not read robots.txt, hits every 6-7 seconds, and is not too smart on what it gets (gets documents twice or three times.)
The robot grabs everything linked (.html,.shtml,.pdf,.jpg ..)
Ideas?