WebStripper

I know, you probably already have it banned...

[carfac]

Hi:

I know.... if you only read 1/10th of the posts in this forum, you already have webstripper banned... and I am sure glad I did.

This THING is so bad, I thought I would just share what it TRIED to do to me the other day!

20 hits as "Webstripper/2.52" in 2 seconds... rejected right off with a UA ban.

20 second pause

comes back as "Mozilla/4.0 (compatible; MSIE 4.01; Windows 2000)" and, in a 5 minute run, tries to grab 2500 pages. Luckily, it got banned three times more, once for hitting a spider trap (on it's 3rd hit!) and then later- while eating my 403's!- for excessive CPU and then bandwidth (all of which banned the IP).

Then, it came back 20 minutes later for another 5 minutes and another 2500 pages, but he was already history from my site.

Just some guy trying to build HIS website from my data...

dave

[wilderness]

Dave,
IMO we're going to see more and more of these attempts. Not just from this products but old ones and new ones as well.

Part of my resources is the chasing down of leads off my referrer fields in my logs (of course less of those is also becoming common.)
Frequently all that is needed is the addition of a key word or two and the searches clean right up.

In the past six months I'm seeing sites and references listed which did not exist in the past. Whether these additions are a growning sign of the internet as SE's just progressing or websites themselves creating more SE friendly pages? I'm just not sure.
I don't take much of the time that I did in the past viewing the meta info on these newfound resources. As a result I may never see the answer to these new things coming online.

[Andrue]

How do you get a ban in on excessive cpu usage?
Thats interesting.

[carfac]

Hi:

Think it is called Stonehenge::Throttle or something.