ActiveTourist gone mad

[Hobbs]

6 page request per second!
Added up to over 600 pages in few minutes
did request robots.txt though,
but will receive a nice 403 forbidden from now on.

Seriously considering a disallow all bots in robots.txt and whitelist the few major ones, then there would be less excuses for irresponsible bot owners, what a pity.

Related threads on ActiveTouristBot
[webmasterworld.com...]

[webmasterworld.com...]

[JustSurfing]

I have contacted the programmers who designed the spider/crawler and am waiting a response to add crawler-delay to code
George