Forum Moderators: DixonJones
216.127.84.21 - - [26/Oct/2003:10:45:40 -0800] "GET /robots.txt HTTP/1.1" 206 1524 "-" "Haste/0.11 (HOME: http*//haste.xatka.net/)"
216.127.84.21 - - [26/Oct/2003:10:45:40 -0800] "GET / HTTP/1.1" 200 20538 "-" "Haste/0.11 (HOME: http*//haste.xatka.net/)" Their website makes a statement to the effect that it crawls a site only once and that it ...does not access any specific Web site at a high rate or volume.
I would hardly characterize the taking of 4-6 files per second as being slow.
Given that 'Haste' purports to visit only once, I'd suggest banning this puppy now before it rips thru your work.
Oh, and if you're lucky enough to have a ban in place against 'Haste', you'll be comforted with the knowledge that they say they obey robots.txt.
Now, isn't that special? A tad late to have much merit, but special none-the-less. Don't you think?
:-\
Pendanticist.