Are there "bot networks"?

Controlled by the same person or company

[Reno]

I use a perl script to identify any bots that deep crawl my sites. Some days there is nothing listed, somedays only a major bot like Yahoo/slurp (most common), but some days I'll check the list and will see that I've been hit with 5 or 6 (or more) bots that are in no way associated with Google, MSN, or Yahoo. It strikes me as odd that this happens in a single overnight time period, especially since the range of IP numbers is spread out. Here's a recent example:

216.246.76.***
38.102.88.***
72.32.210.***
74.52.112.***
81.251.184.***
82.231.31.***
83.190.126.***

Is this a coincidence, or are there bots being launched in a coordinated fashion, from multiple locations?

Was wondering if anyone else has noticed this? (or maybe I'm getting paranoid)

........................

[edited by: volatilegx at 5:14 pm (utc) on Feb. 3, 2008]
[edit reason] obfuscated ip addresses [/edit]

[wilderness]

216.246.76.*** hosting
38.102.88.*** Performance Systm
72.32.210.*** a colo
74.52.112.*** ThePlanet

These are all pests.
If they spidering your pages without a proper protocol of identidying themselves?
Why allow them to continue?

There's certianly no benefit to your websites from such pests.

Somebody else will need to answer your inquiries on the RIPE ranges.
I don't monitor them.

[Reno]

I do use htaccess to deny them, but unfortunately that doesn't happen until after they spider that one time (so my script can pick up their IP). It may be coincidental that I'll go a few days in a row with only one or more of the big 3 coming by, then suddenly it will be 5 or 6 new/unidentified botpests all in one night. Seems kinda' weird, and made me think that perhaps it's the latest attempt by the spambot creeps to get by the blocks that people are putting in place.

..........................

[wilderness]

I do use htaccess to deny them, but unfortunately that doesn't happen until after they spider that one time

I'm assuming (as presumptious as that may seem) that your script is automatically denying to the precise Class D range?

As a result, the bot merely reappears as a new visitor (at least to your script) from a new Class D (or even Class C in some instances) range.

If so?
A more effective solution would be to monitor and modify each script change to the inlclude the FULL IP range of the abusive provider.(thus implementing a long-range solution).

It may be coincidental that I'll go a few days in a row with only one or more of the big 3 coming by, then suddenly it will be 5 or 6 new/unidentified botpests all in one night. Seems kinda' weird, and made me think that perhaps it's the latest attempt by the spambot creeps to get by the blocks that people are putting in place.

Believe I had a dry spell last year where it was nearly three months before a change was made in my htaccess to include a denial and then whamo, an assortment of new ranges appeared.
The possibility of a big conspiracy is not too far-fetched, however I'm more inclined to believe its just coincidence

Don

[Reno]

Thanks wilderness. Your assumption is correct and your suggestion is one I'll need to implement. I want to deny the bad guys but do it in a way that would block no more regular visitors than is necessary to accomplish the goal -- which to my non-expert eyes seems like a fine line to walk, so I tend to proceed with caution.

..........................

[wilderness]

216.246.76.*** hosting (the subnet range is merely a reseller underneath the backbone)
RewriteCond %{REMOTE_ADDR} ^216\.246\.([0-9]¦[1-9][0-9]¦1[01][0-9]¦12[0-7])\. [OR]

38.102.88.*** Performance Systm
RewriteCond %{REMOTE_ADDR} 38\.102\.88\. [OR]

72.32.210.*** a colo
RewriteCond %{REMOTE_ADDR} 72\.32\. [OR]

74.52.112.*** ThePlanet
RewriteCond %{REMOTE_ADDR} 74\.(5[234])\. [OR]

Note 1) (last [OR] should only be included if it's followed by additional lines)
Note 2) Make sure to correct forum breaking of the pipe characters before use.

Jim might advise you of the exception in the Performance range for gigablast, otherwise, there's not a SOLITARY innocent in the exclusions above.

Denying short will always result in a bite taken out of your own backside.

[Reno]

Thank you again Don -- that's a tremendous help.

If I may ask one more general question...

When a person denies using the broader technique as you have generously explained (as opposed to the specific one I was using, based on the captured full IPs), is there a general rule-of-thumb about how many users are being affected?

In other words, if I'm blocking one user with a full IP (which is less effective), will the broader but more effective "deny" cut off 10 additional people? 100? 1000? Just wondering if there is a generally accepted number that could be considered close to accurate, or does it entirely depend on circumstances (such as location)?

......................

[wilderness]

In other words, if I'm blocking one user with a full IP (which is less effective), will the broader but more effective "deny" cut off 10 additional people? 100? 1000? Just wondering if there is a generally accepted number that could be considered close to accurate, or does it entirely depend on circumstances (such as location)?

Reno,
We have no way of knowing if that focused Class D is a shared or fixed IP!

In most instances, these colo or hosts are simply providing IP racnges for sale to websites!
NOT proving internet service to actutal mainstream internet customers. (are your websites looking for traffic from other websites [harvests] or actual real-live people?)

You may rest assured that if you allow "some" (for lack of better words) of these harvesters in, that lack the protocol of providing their identity, that word will spread and more harvesters will be passing through the doors of your websites as well.

All four of these rewrites that I've provided have been mentioned numerous times in Forum 11, with perhaps the only exception being the 216.246. range.
Thus your "going along" would generally be construed as joining the masses ;)

In summary, although these may seem extreme numbers to you?
Any denial of service is simply the back-bone providers own fault for not enforcing a firm UAG that has the possibility effecting their entire network. (most UAG's contain such a clause regarding network threats).

In addition, were the back-bone provider with a conscience?
They break their service ranges (via registrar; i. e., ARIN or similar)into subnets which would expose their users name.
EX: 216.246.76.***
1) Provides the name of the back-bone
2) the name of the reseller
3) NEGLECTS to provide the name of person (s) that is the resellers customer (actually doing the harvest).

You might try contacting some of these folks and understanding their point of view

("we haven't done anything wrong", "100 pages, 200 pages? So what", "Protocol, identify myself, bite me").

Guarantee that you'll change your short-denying procedures promptly.

Don

[blend27]

-- but unfortunately that doesn't happen until after they spider that one time --

That is why we have bot traps that are disallowed in robots.txt, dynamic/random bot traps(no robots.txt required), dynamic robots.txt that if requested and NOT Coming from a major SE gets IPs blocked. All kinds of fun! Ever thought of Random Redirect?

[Reno]

Thanks again -- the more I read at this wonderful forum resource, the more I realize I do not know. So much to learn, so little time....

..............................