Anyone know about "Lite Bot 0816b"? Skipped robots.txt and went straight for my sitemap. Then downloaded the whole site. Below is the first entry of my log.
Lite bot TRIED to hit last night- got nowhere but the door!
Thanks for the heads up!
dave
mvl22
12:16 pm on Nov 28, 2002 (gmt 0)
I've banned it. It generated overnight a pile of 404 errors in quick succession, so I regard it as potentially hostile.
dmjw01
10:03 am on Dec 21, 2002 (gmt 0)
I got hit by this yesterday. It didn't bother with my robots.txt file, but didn't get any further than my root page - I *knew* there was an advantage to frames! :)