Forum Moderators: open

Message Too Old, No Replies

Spider ID?

Thunder stone

         

wilderness

11:21 pm on Jul 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Haven't seen this creature in a while.
Honored robots.txt

206.183.1.74 - - [18/Jul/2005:15:38:51 -0700] "GET /robots.txt HTTP/1.0" 200 3408 "-" "Mozilla/4.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)"

silverbytes

1:22 pm on Jul 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I banned that yesterday, but If I'm wrong please post. I don't belive a decent spider uses a name using a name like that.

Angelis

2:19 pm on Jul 25, 2005 (gmt 0)

10+ Year Member



Information about it here. [webmasterworld.com...]

GaryK

3:43 pm on Jul 25, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



On my servers this bot reads robots.txt and then promptly ignores it. It's been banned for years ever since I caught it reading pages I disallowed in robots.txt. Also, and maybe more significantly, they don't appear to offer me anything in exchange for spidering my sites and that alone is enough to get even a legitimate spider banned. In the end it's a decision you have to make. :)

BTW, a spider named Webinator also eminates from the same company and I ban it too.