"HTML Text Download Class" Agent

Does not respect robots.txt

[carfac]

Got hit by this one last night- from Instituto Tecnologico y de Estudios Superiores de Monterrey ...

Didn't take much, because it ignored robots.txt and got itself banned mucho pronto!

Started off like a 'bot- requests for all the links off my top page in a second, but then I see requests every 5-10 seconds, like a real person, so I guess it might be a web accelorator (which I bann just because...)

148.233.159.250 - - [24/Sep/2002:04:56:13 -0600] "GET /search.cgi?query= HTTP/1.1" 200 23818 "-" "HTML Text Download Class"

dave

[Brett_Tabke]

I've seen that agent myself and trying to connect it with a program. Anyone know?

[carfac]

Hi Brett!

Sorry- I have not been able to turn up anything...

dave