homepage Welcome to WebmasterWorld Guest from 54.196.159.11
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
IP number for countries
APNIC souce, others needed
privacyman




msg:893458
 2:43 am on Apr 29, 2003 (gmt 0)

I did find not long ago a source for listing IP numbers based upon the typical two letter designation for countries. This was found to be through APNIC where a person could specify AU or another APNIC member country and get a list of IP numbers for that desired country.

[apnic.net...]

displays this data for a given ISO3166 country code.

The data available is only applicable to IP/ASNs from the APNIC ranges, ie, if the IP range was allocated pre-APNIC, it is not in that list at the present time.

This I had found to be very handy whereas I wanted to block a couple of countries for email and/or web, but I did not want to block others or the entire APNIC group.

Is anyone aware of any other registries, ARIN, RIPE, etc that may have similar functions available so that one could list by country code?

When looking up a particular IP such as 123.4.56.79 it might show results of XYZNET-21 as 123.4.56.128 - 123.4.56.255
that would show the range that the individual IP is within and the provider name. I learned you could then search for XYZNET without the addon, usually to find ALL IP's by that company. But sometimes the list is long, and may not be in a numerical order for IP numbers. Any way to specify to those registries to show the results with IP's in numerical order?

And last question. Can someone clarify to me the meaning of an IP group when it might be shown as 123.4/19 for the above, but this example may not be technically correct? It's the /number that puzzles me for the range that they mean.

Thanks.

 

dkubb




msg:893459
 6:37 am on Apr 29, 2003 (gmt 0)

Hi,

There is a perl module on CPAN called Geo::IP that comes with
a free database that maps IP address ranges to countries.

You can download the database (in DBM format I believe)
from the following URI:

maxmind.com/download/geoip/database/

[edited by: heini at 11:05 am (utc) on April 29, 2003]
[edit reason] delinked [/edit]

PsychoTekk




msg:893460
 10:42 am on Apr 29, 2003 (gmt 0)

sounds nice, but what do i have to do to use this mod under win32 apache?

indiandomain




msg:893461
 10:50 am on Apr 29, 2003 (gmt 0)

i was trying to find the ips of each international city.
is this possible to find?

indiandomain




msg:893462
 10:50 am on Apr 29, 2003 (gmt 0)

i was trying to find the ips of each international city.
is this possible to find?

bonanza




msg:893463
 10:51 am on Apr 29, 2003 (gmt 0)

I use that geoip perl module (a godsend).

I found it to be a little quirky on *some* windows boxes. I didn't need to figure out why, so I don't have much detail.

I use a straight perl program to filter a tab delimited file and add the country column at the end.

It's pretty simple to use, so if you find you can't get your script working correctly, give it a quick try on another machine.

Gorilla




msg:893464
 10:55 am on Apr 29, 2003 (gmt 0)

You won't be able to find IP-addresses for cities in any reliable way. The closest you can get is countries, but even there you can not always be certain.

This as ISPs are allocated IP address ranges which they then allocate to customers. Most ISPs are operating within a country or a region.

indiandomain




msg:893465
 10:56 am on Apr 29, 2003 (gmt 0)

maxmind does have a geoip for cities.
300$ though.not free
:-(

bonanza




msg:893466
 11:05 am on Apr 29, 2003 (gmt 0)

I didn't have to pay for the module. Search around for geoip.

A fresh database might cost money though. I don't know if the author keeps up with it (or if there's even much to keep up with.)

Gorilla




msg:893467
 11:06 am on Apr 29, 2003 (gmt 0)

Are you looking for a list of addresses/masks for a city or are you looking to determine which city a given IP-address is located within? The latter is what I guess maxmind is providing. (Having had a 30 sec look at their page.)

The informasjon provided by GeoIP and maxmind provide is based on informasjon from the whois databases of the regional internet registries like RIPE NCC (www.ripe.net). The informasjon in their whois databases is input by the various ISPs and the quality of that information varies quite a lot.

[edited by: Gorilla at 11:07 am (utc) on April 29, 2003]

heini




msg:893468
 11:06 am on Apr 29, 2003 (gmt 0)

There are other commercial offers to do this.
The basic GEO::IP mod from maxmind is free. I'm not so sure on reliability though.

privacyman




msg:893469
 3:00 pm on Apr 29, 2003 (gmt 0)

Thanks dkubb

for the info about geoip. I will have to get it and check it out by uploading it to my server/host where perl is available to me, don't have perl or equiv on this win98se machine.

The link that I did find at APNIC has proven handy for the APNIC group. Using its capabillity by country I was able to print a list of IP's for AU, NZ, and other countries that I wanted to allow. Then within the appropriate groups I included just about everything else except those that I wanted. Used this list written in regex for blocking of spam from many APNIC countries. And I haven't decided whether to use this same list or similar for blocking of site visitors for a personal site, and three business sites that I manage.

In the case of email, or business, I don't do any communication with most APNIC countries and likewise the sites aren't intended for that audience.

The desire to find other sources of listings for countries of IP numbers is needed for RIPE/Europe areas, and others, so that again the biggest offending providers and/or areas of spam can be further reduced. Also for improving web sites, as I had found some bandwidth theifs in a few areas.

Thus far, much less spam. Doesn't even get to my inbox. Thus no accidental openings, and less trash and time. The websites also have much cleaner logs and more accurate stats. I asked myself why some of the areas would even want to index and or cache some of the pages and images when the audience is not intended for some areas. By some of this selective blocking, of "some" big blocks and areas, it also affords more security for sites, plus more protection against piracy or copyright infringements.

To list by cities is not really necessary. Normal lookups will usually reveal that, but to find all applicable or desired countries, boy, the numbers sure are scattered. APNIC at least has a fairly good group of chunks together.

Thanks. I will check out geoip also to see if it helps.

privacyman




msg:893470
 2:10 pm on Apr 30, 2003 (gmt 0)

dkubb and others....

As for geoip, I downloaded its archive, extracted, and from what I can decipher I would need to have perl to use it on my win98se computer (don't have perl). My other thought was to put it onto my host/server but I decided not to whereas from what I could tell it uses a data table that geoip has at their site.

Searched an engine for "country ip", many results, most for resolving ip's but we already have whois. Found one source which looks like what I was seeking, alike that used at APNIC.

Denmark site at [ip.ludost.net...] seems to have a tool "online" for searching by country, eg to find all IP's in GB, or IT, etc., various options including RIPEDB format (more "readable" for me).

Also an ascii file, at top of page, called "country.db.gz" (it's a unix style compressed file, 391k; its text file is 1.8m) which lists all (or probably most) IP numbers, starting with:

0.0.0.0 2.6.190.55 -
2.6.190.56 2.6.190.63 gb
2.6.190.64 2.255.255.255 -
3.0.0.0 4.17.142.255 us
4.17.143.0 4.17.143.15 ca

View source of that tool page, looks like a "safe" page to use. Did check for any spam complaints, for ludost.net and found at google abuse group about 88 entries so they may have newsgroups and a few problems there.
[groups.google.com...] [if inappropriate, please kill]

Anyone else have opinions on this tool source? From what I can see I think it's safe to use as long as one doesn't get on their newsgroup lists.

With both the lookup capability by country, as well as a list by number showing ranges and countries per range (example snip above) this is almost more that what I was seeking. This way, lots of bad habits for web visitors or lots of spam, say from 80.x.x.x group, at least now I can decide more easily whether to kill a small range or a larger block and know what's within it and selectively make a better educated entry with regex blocking.

Would appreciate opinions. Hope this also turns out to be a good tool that others can also use for combating various abuse sources.

indomitable




msg:893471
 10:27 am on May 2, 2003 (gmt 0)

Isn't this rather academic anyway? After all a huge number of visitors are going to come from dynamically assigned IPs, owned by ISPs. A huge number of visitors will APPEAR to come from Virginia because they are AOL users etc.

privacyman




msg:893472
 10:58 am on May 4, 2003 (gmt 0)

Welcome to Webmasterworld indomitable

You wrote: Isn't this rather academic anyway?

Academic purposes can be beneficial, a lot depends upon evaluation and how it can be used. Both in regards to email and web sites, whois functions have been very good tools towards stopping spam and idenifying code or property (theft) and email parsing robots. Whois also has been an aid in identifying companies, but most usually it shows the general location of an ISP where an email or visitor originates from.

In the case of AOL users, the main office is located in Virginia. In most cases with AOL or other large ISP's they have smaller IP groups allocated to certain areas. The actual IP looked up will usualy show the smaller local area (s.west, n.west, NY, or similar) and the parent IP group would show the main office location such as VA for AOL.

For marketing of a web site it can be beneficial to know the general areas that viewers come from. Many hits from Calif or southwest and nothing from north mid-west, could indicate season/climate reasons for lack of visitors from one area, or that a product is of no interest in that area. Thus, web sites can be improved.

For usage related to email. Whois has been very beneficial in identifying spam and fraud sources. Within the header of emails, the (usually first or topmost) "received from" portion shows the IP number that my mailserver received the email from. Subsequent, or additional "received from" portions may show a dialup IP or other source, or may be forged.

After having studying those headers (especially the entry made by my mailserver), documenting the sources, and evaluation, I found that certain countries, areas, and/or providers were major sources of spam. As a result I chose to block email from those areas for both personal and business email. My thought? Will I be doing much sales if any to those areas? Do I normally communicate with anyone in those areas? Whereas it is "our" email program that is receiving much unwanted spam, as the receipient it is also "our" right to stop it. [Note: Televised on Wed, Apr 30th, the FTC and others were discussing better controls on spam and laws to stop it. Until such time that they are effective I shall take my own steps.]

As a result of increased spam, I've study web logs for personal and business sites that I manage. It was found that some bots and access was coming from the same areas as much of the spam. Whether related or not (some may have been email harvesters, others may have been directories), I had to determine companies related to IP's, or source areas. Did I intend for or need exposure of these web sites in those areas? Would any business be transacted with those areas? (These business sites that I manage have no desire or intent to do business in APNIC areas.) Would the personal sites be of use or interest to people in that area? (The sites area in english language only.) A choice was made to block those areas thus reducing server load and spam.

Other reasons (than spam) for blocking bots, directories, and visitors from some areas was to reduce or eliminate page mirroring, code theft, image theft, and/or bandwidth loading. Most content on these sites is copyright protected. Unless a person is a large corporation with lots of time available, the lawyers, and the dollars for legal action, the oridinary persons can not afford to fight copyright infringement. There are differences in country laws and most don't have to respect the laws of another country.

The multiple sources of whois has been very valuable towards identifying sources of spam as well as identifying visitors, bots, and (possible) theives. Use of whois along with country lookups (to see what IP's cover an area) have both been useful in controlling many of these negative effects (also helping to keep from blocking other areas that are not a cause of problems), and they have been an aid in determining what improvements might be needed for sites that may have something to offer an area.

Hopefully if the FTC and other agencies create tighter laws against unsolicited email, fraud, and other illegal acts, maybe at that time the harvesting bots will disappear along with a lot of the other negative effects that are associated.

Most definitely, I and others value our time, our creations and our property, and we definitely have the right to protect any and all of those things. And as a member of the FTC meeting said, the first amendment was not intended to give a right to sell something.

In answer to your question or comment of it being rather academic, yes, it is educational, informative, and helpful in many ways.

privacyman

indomitable




msg:893473
 12:10 pm on May 6, 2003 (gmt 0)

Yes, I understand why you are interested; indeed you are doing all the things I imagined you might be.

My point about it being academic ( in the "Theoretical or speculative without a practical purpose or intention" sense) was simply that I wondered if this could be useful to you if it was inaccurate. You have actually reassured me that these IP lookups are probably more accurate than they I had thought.

privacyman




msg:893474
 8:31 am on May 7, 2003 (gmt 0)

indomitable,

As for the accuracy of whois lookups, whether for checking for sources of spam, or source isp's for dialup to web site, or for the origination of bots coming to web sites.... the accurary of the whois lookup would probably depend upon how soon the lookup is done past the day of incident. If it's a week later and one is checking the IP of spam, an offender could have been terminated, but most usually IP numbers remain valid for at least a while with the registries (ARIN, RIPE, APNIC, etc).

If it's a bot that is associated with an IP that you're checking or a referring site that is trying to hide its domain name, if an ISP terminates the account holder, I would think that an ISP would try not to reassign a dedicated IP number to a new customer for at least a short duration (in order to elimate possible problems... sort of what telco's do with phone numbers, they wait at least a short while before giving a number back out to a new customer).

In the case of spam, if there is just one "received from" entry in the header then that is the sender. If there is more than one
"received from" then it is the topmost one in which is might show alike

Return-path: <sender@somedomain.net>
Envelope-to: recipient@adomain.com
Delivery-date: Tue, 06 May 2003 07:26:19 -0700
Received: from a.smtp-out.sonic.net ([208.201.224.38])
by mymailserver.com with smtp (Exim 3.36 #1)
id 23D3Oc-0001cc-00
for recipient@adomain.com; Tue, 06 May 2003 07:26:18 -0700
Received: from sender (dial-209-148-114-73.sonic.net [209.148.114.73])
by sub.sonic.net (8.11.6p2/8.8.5) with SMTP id h23EQJV08843
for <recipient@adomain.com>; Tue, 6 May 2003 07:26:19 -0700
X-envelope-info: <sender@somedomain.net>
Message-ID: <000231c313db$5bdbc180$497294d1@sender>
From: "Mary Doe" <sender@somedomain.net>
To: "John Doe" <recipient@adomain.com>
Subject: Usually a valid topic here

Not all headers look alike, but are similar. Not all will have an Envelope-to or X-envelope-info, and some may have additional items. In this example, modified for privacy, with 2 received from portions, the topmost was added by my mail server. That is a valid entry ALWAYS! That is who my mailserver got it from and it cannot be forged. The second one here is a valid dialup IP, used by the sender.

Variations. The one received by my mail server could point to the sender's mail server, as in this case, or it could be a relay point. Either way, it is still valid. The second entry could be real or forged, and it could be a dialup, or a relay point. In most cases for spam, if more than one received from entry, the subsequent entries are either forged {with hopeful intent to confuse John Doe receipient as to who the real sender is) or the subsequent entries are irrelevant.

In cases of spam, I watch only the topmost that is an entry provided by my mail server. If too many spam from one ISP or one "area" (eg, if from Antarctica), or one company, then I evaluate if they have a "problem" of hosting too many spammers and if I need email from that source. The decision to block or not is then made.

Once a person gets into the habit of checking headers of many spam, you learn how to determine what is forged or real for the "subsequent" received from entries. In this example, both have the ascii id of the host for the ip numbers and the second one indicates that it is a dialup too and lookups are accurate and match.

Usually for the little spam that I now get, I still keep a brief cumulative list of copy/paste of the headers as new spam comes in, and at end of day do the lookups on those few, and every so often sort that list to see if I need to kill some source, whether it's a relay point, a country, or a particular provider. Alternatively, I might decide to contact an upstream provider if the a spam source is a particular ISP or domain so that they can possibly terminate the offending client.

Research of IP's for bots, or if I see a lot of hits coming from a particular IP number, is usually fairly easy to deal with. Depending on the web log info, IP, referer, user-agent, and action being requested (GET of files, images) or other activity, one can learn from studying the log entry and doing a lookup. Depending on all the factors and lookup, one can determine if it's a visitor or a bot (for SE) or if it's some kind of "harvester".

In answer to your being curious about accurary, I would have to say that for most of the time the whois lookups are usually quite accurate. The topmost entry of an email header cannot be forged. And as far as I know, the IP number shown leftmost in a log file also cannot be forged. In the case of spam, in order for it to be totally combatted, if all ISP's and valid relay points recorded originating IP numbers (may actually be, or could be possible, currently unknown to me) then all spam could be traced to a true origination (dialup or dedicated). Easy to implement if not done. Then any forged info in an email would be of no use to any fraudulent senders.

Now, as an ex-telco man (circa 1980's) I know that even local phone calls are documented. A fact kept quiet and not known by many. If one considers fraud and other illegal activities via the internet, via email, or via phone.... well, if IP numbers were all traceable to originating IP (dialup or dedicated line), with logon being required, add in the fact that local calls are documented in central offices on their c.o data tapes, all it will take is some new laws to stop spam and enforcement of those laws and existing fraud laws. The means for legal agencies to "find" offenders is here, and it has been for many years.

Ever see the movie "War Games" (Matthew Broderick, 1983)? It did portray a bit of realism.

Hope this helps with your knowledge about IP numbers, how they're used, can be used, and how some try to use them.

Regards

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved