This appeared the other day in my logs and grabbed loads of pages including the robots.txt file. Anyone know what it is?
Thanks!
allybongo
9:49 am on Sep 3, 2002 (gmt 0)
Sorry, it's coming from 195.217.192.34
SmallTime
10:21 am on Sep 3, 2002 (gmt 0)
An offline browser utility called HTTrack that downloads entire sites.
Kerrin
8:14 pm on Oct 3, 2002 (gmt 0)
I was hit by the same thing today. It did request robots.txt but ignored the no robots crawl bit of it. Luckily it triggered an abuse script and was automatically blocked from my server before it got many pages.
ratman
8:48 pm on Oct 3, 2002 (gmt 0)
I was hit several times by this one before I worked out how to block it.
