Problem with class C Ip Checks

[johnhamman]

Hi All, Question. If i take the ip address of the spider visiting, then knock off the end if it and make it a class c ip. ( 204.152.191.28 to 204.152.191) then do a search in the database it will flag that its a spider because of the folowing ips:204.152.191.41,204.152.191.47, 204.152.191.57, and 204.152.191.58. But Its altavistas translator service!
so whats the best way to get past this?

my pseudocode is:
function SpiderVerify(ip,UA)
'ip = ip address
'ua = User Agent
make ip class C
'123.123.123.123 to 123.123.123
check to see if any ips like that in database.
flag good for no matches.
flag bad for matches.

[johnhamman]

Also why make it a class C instead of the full IP?

[Air]

The advantage is that spiders tend to be added from the same class "C" IP range, so it provides an opportunity to detect spiders that would otherwise be missed.

[volatilegx]

You should keep a database of translators as well. Do a check on the translator database before the spider database. Here's the IPs I have listed as translators:

204.123.9.65
204.123.9.66
204.123.9.67
204.123.9.68
204.123.9.106
204.123.9.107
204.152.191.27
204.152.191.28
204.152.191.29
204.152.190.27
204.152.190.28
204.152.190.29
204.152.190.37
204.152.190.154
204.162.96.104
204.162.96.154
204.162.96.176
209.247.194.35
209.247.194.100
64.208.35.5

[johnhamman]

Wow, Thanks so much, this is what i do:
first i grab the ip address and UA. then i also make the ip a class C. now what I do is check it against a sql2000 database and it does this check.
is IP a translator yes/no
if no
is class C ip like any spiders yes/no
if no
is UA like any spider UAs

This is as thurrow as i can get i think. Anyone else have any better ideas?

[volatilegx]

I would do the check against the list of translators before truncating the IP address into a Class C.