Welcome to WebmasterWorld Guest from 54.242.53.253

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Definitive Sources of IP Lists

     
8:03 am on Nov 28, 2012 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


Been looking for some definitive IP list sources maintained by the owner and not guessed at by 3rd parties by observation and a couple I came across are:

Microsoft's Azure Cloud Services provides an XML file:
[microsoft.com...]

Google provides a simple list in DNS from the command line:
nslookup -type=TXT _netblocks.google.com


Which returns:
_netblocks.google.com text = "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all"


Anyone else got any direct IP list sources like these for Yahoo, Bing, etc.?
12:39 pm on Nov 28, 2012 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 13, 2005
posts: 128
votes: 0


That's a great list - thank you for sharing.
7:29 pm on Nov 28, 2012 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


Thought I'd add the IP assignments per country for those new to access control that don't know about these:

ftp://ftp.apnic.net/pub/apnic/stats/apnic/delegated-apnic-latest
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-latest
ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest
ftp://ftp.ripe.net/ripe/stats/delegated-ripencc-latest
ftp://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest

Anyone got anything else good, anyone?
9:48 pm on Nov 28, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3125
votes: 4


The MS lists are (mostly) very short ranges! A real pain to add to blocklists. Probably worth doing, though, if they are clouds.

The v=spf returns are for mail (part of DKIM encoding/checking) - they specify IPs that can send mail so are probably not a threat to web servers.

Update:

Looking more closely at the MS ranges: some can be concatenated; others I already have listed as bot IPs. Does that mean they are changing usage?
11:40 pm on Nov 28, 2012 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


From what I read the Google SPF lists are pretty all encompassing because the change IPs for stuff all the time and that's pretty much the list I had too. Says right on their website that it's where to get the definitive list of IPs and that comment was in the Google Developer section I think and wasn't an email discussion.

MS IP usage is all over the map so who knows. I think any IP is fair game at MS to be used for anything.

I just want sources that I can download and process automatically and keep updated without going stale and these look good so far but you're right, the MS list needs some concatenation.
8:38 pm on Nov 29, 2012 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3125
votes: 4


It figures that G would abuse the spf system. :(

I've been wanting to ban G mail for a long time but my customers and their customers still insist on using it.

A while ago I began blocking all "bot company" IP ranges except for known bot IPs. This seems to block some "mobile" proxies but since Y and G seem to use these IPs indiscriminately, tough.
1:44 pm on Dec 1, 2012 (gmt 0)

Junior Member

10+ Year Member

joined:June 25, 2005
posts:179
votes: 1


From the Azure XML file:
<network>65.55.192.0/19</network>

Now I see "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)" coming from 65.55.211.186 <=> msnbot-65-55-211-186.search.msn.com. This Azure IP list is garbage.
2:38 pm on Dec 1, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1851
votes: 49


I started using BGP Toolkit from Hurricane Electric site for finding IP Ranges that belong to specific organization/colo/datacenters a while back. These are on more granular level. Seems to work OK.

Goog IPs for example: bgp.he.net/search?search[search]=google .
---------------------------------------------------------------

The RIR lists from FTP Sites that incrediBILL mentioned are also pulled on monthly bases and processed into a DB. Also have a subscription for ip2location DB.
8:42 pm on Dec 1, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 26, 2006
posts:1619
votes: 0


Interesting stuff for sure. I use ip2location as well, but maxmind has a really great one as well that I use on an ad need basis. You can look up 25 per hour free on their site.
GeoIP City
GeoIP Country
GeoIP Region
GeoIP Organization (I especially like this one)
GeoIP ISP
GeoIP Netspeed
GeoIP Domain Name