webmasta - 8:45 pm on Dec 6, 2002 (gmt 0)

back to the bot list ... i am new here (first post) and found this thread very interesting so i tried it on my site/

i downloaded the new version 4.37 of BlackWidow ... and used the last updated list that superman posted / somehow BlackWidow is getting past the htaccess file and downloading the site. However when i tried it from http://www.wannabrowser.com/ and changed the UA to BlackWidow it was blocked .. getting the 403 from there /

any takes on that? this new black widow comes with a ton of plugins that could get around almost anything including encrypted sites! and can decode script generated url's....

Next .. i noticed "Web Spider" in my logs. Any idea who this is?

Also .. while searching for some offline browsers i came across www.matuschek.net/software/jobo/index.html .. seems like this JoBo is a smartass / it gives u the option to change the UA to anything u want! of course when i added JoBo to the htaccess file it was blocked but when i changed the UA to Mozilla it downloaded the entire site!

This one is dangerous! guess we might as well start thinking that insted of blocking unwanted bots we only allow the good ones.... its just as hard or even harder to keep on top of the bad ones and add them to the list as keeping on top of the good ones and add them /

webmasta

[1][[b]edited by[/b]: jatar_k at 9:04 pm (utc) on Dec. 6, 2002][/1]