Forum Moderators: open

Message Too Old, No Replies

arste.info bot

         

Pfui

9:46 am on Aug 28, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



nnn.dip.t-dialin.net
Mozilla/5.0 (compatible; arste.info_bot/1.1; +http://www.arste.info)

robots.txt? NO

(Yet another iffy dipster.)

keyplyr

8:35 am on Aug 30, 2009 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



After reading your post, I checked out their info page and added a URL of a site I manage. It crawled entire site a few hours later taking 10 files per second. And since it did not get robots.txt it took many disallowed files.

80.143.119.** - - [29/Aug/2009:04:44:38 -0700] "GET www.example.com/ HTTP/1.1" 200 11002 "-" "Mozilla/5.0 (compatible; arste.info_bot/1.1; +http://www.arste.info)"

Banned until they get their act together.