Forum Moderators: DixonJones

Message Too Old, No Replies

List of bad bot/spiders

Requesting a list of known bad bots

         

clembot

12:29 am on Sep 7, 2005 (gmt 0)

10+ Year Member



Does anyone have a current list of bad bots to block. I am seeing more and more bots which I have no idea whether should be blocked. For example:

Libby_1.1/libwww-perl/5.76
ichiro/1.0_(ichiro@nttr.co.jp)
dlman
Wildsoft_Surfer
Mozilla_blah
nicebot
Sleipnir_Version_1.60

I tried looking around the forums and googled them, but can't decide which ones should be blocked.

Any help would be appreciated (either on the specific ones listed above, or someones current list of bad bots).

Thanks!

physics

2:59 am on Sep 7, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Check out A Close to Perfect .htaccess Ban List [webmasterworld.com] and the children of that thread.

clembot

11:19 am on Sep 7, 2005 (gmt 0)

10+ Year Member



Thank you for the reference. I actually did see that thread before but thought it may be a little out of date (it ranges from 2000 to 2003). Does it still apply or is there a newer list available?

jatar_k

8:35 pm on Sep 8, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



the child threads go all the way up to 2004 and, for the most part, those threads are pretty inclusive and current.