Page is a not externally linkable
wackybrit - 10:51 pm on Oct 30, 2003 (gmt 0)
And the best have vast networks so they can appear to be coming from 100s of different IPs, with different UserAgents every time.. much like regular visitors. I'm not sure if there's a viable way to block this sort of activity, as it's very hard to track.
A "good" leeching bot (in the eye of the leecher) will disguise its UA and never obeye a robots.txt.