Forum Moderators: open
These folks just sucked down 32,000 pages in one day
- the site only has 300 or so pages/images they just scanned them repeatedly.
Almost 200 megs in 12 hours or less.
Quite evil.
Any reason why I shouldn't block the entire 38.x.x.x PSI range?
[edited by: volatilegx at 2:17 pm (utc) on Sep. 14, 2006]
[edit reason] removed some specifics and link [/edit]
For this one in particular; go to ARIN and copy the following without the quotes:
"> 38.99.203."
Then scroll down the page and you get them all that surrond the Class C in this example.
NOTE; to those that like to play around.
Some of these subnets can be rather large and take a long time to load up (no matter how fast your own connection is). I would suspect that abuse of these types of searches would eventually cause you to be denied access.
George had similar problems with too many referals from his website.
BTW, there's a way to do subnet delegations at RIPE also, however I've never been able to figure it out.
But since PSI owns the entire 38.x.x.x block, unless someone can show me a useful service that exists in that range, I'm going to block the entire thing.
PDS's bot was extremely abusive. Repeatedly scanning, ignoring robots.txt and sucking down over 200 megs in 24 hours (to start with).