homepage Welcome to WebmasterWorld Guest from 54.234.0.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
International Data Centers
incrediBILL




msg:4609692
 2:00 am on Sep 14, 2013 (gmt 0)

I find myself having difficulty sorting out what's a real data center vs. non-commercial IPs in places like Russia and China.

Some things are obvious but most of it leaves me scratching my head.

Anyone got any tips or pointers about sorting out the IPs from RU and CN?

Would be helpful.

TIA.

 

lucy24




msg:4609703
 2:25 am on Sep 14, 2013 (gmt 0)

I assume you're looking for something more tightly constrained than "Kill them all and let {deity of your choice} sort 'em out" :(

dstiles




msg:4609817
 9:41 pm on Sep 14, 2013 (gmt 0)

I take a few random IP scans using umit. If x samples return no open ports then it's probably DSL, although occasionally I get a false report for a cloud using this technique. NOTE: A single scan on an IP taken from a "security" log is not conclusive as virus-contaminated IPs usually show open unless the computer has been switched off.

Probably a secondary indication for people in the US is that EU-area DSL machines are usually turned off during your "daylight" hours. Not sure how this applies to china - I'm not that good at time zoneds. :)

I've heard it said that IP scans are evil and anti-social but in my book if someone hits my server with a blockable access then fair game.

incrediBILL




msg:4610013
 6:40 am on Sep 16, 2013 (gmt 0)

Some things are pretty obvious, esp. for those that return reverse DNS data.

Guess my real problem is trying to make sense out of some of the APNIC records.

Some countries are pretty easy, others are a real PITA, especially if you don't know some of the local terms and such. Although I've been doing this for many years I still find it a real learning experience and often I find Asian countries the hardest as they don't deliver the data as granular as needed and/or using the terms required to make simple decisions.

I've resorted to blocking entire providers (like a Comcast equivalent) or worse case entire countries just out of frustration.

I take a few random IP scans using umit.


Do you do this via a proxy or directly from your server's IP?

Without using a proxy, I'd be worried about having my IP flagged as a potential security risk for scanning people's IPs.

bhukkel




msg:4610050
 10:53 am on Sep 16, 2013 (gmt 0)

Guess my real problem is trying to make sense out of some of the APNIC records.


The problem i have with APNIC records is that they are not so structured as the RIPE or ARIN records. So processing them with PHP is a lot more difficult.

dstiles




msg:4610152
 7:31 pm on Sep 16, 2013 (gmt 0)

Bill - are you using linux? If so, umit and select Quick Scan - I find that's adequate.

No, I do not use a proxy even though I have a fixed IP. I do not make that many scans on any given network at any one time and I've never detected a rejection (that I know of!). I suspect that ISPs do not have the time/patience/resources to worry about the odd IP scan when they must be bombarded with botnet/etc scans continually. And servers get a LOT of those!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved