homepage Welcome to WebmasterWorld Guest from 54.167.182.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Iparadigms
keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4358078 posted 7:24 pm on Sep 2, 2011 (gmt 0)

38.111.147.86 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) -"

rDNS: no
robots.txt: no

Went for HTML files only, no images,scripts,etc

Anyone blocking all of Cogentco?

38.0.0.0 - 38.255.255.255
38.0.0.0/8

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4358078 posted 9:23 pm on Sep 2, 2011 (gmt 0)

Not all but several sub-ranges.

I have 38/8 listed under PsiNet Inc but yes, it's cogent.

I've become very harsh on MSIE-6 recently. A suspicion of bad headers and Bang! Apart from MSIE-6 being deprecated by MS, they also no longer patch Windows 2000 (NT 5.0) so that's another bad mark.

Pfui

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4358078 posted 11:35 pm on Sep 2, 2011 (gmt 0)

I've redirected Cyveillance, and PSI, and goodness knows what all with --

RewriteCond %{REMOTE_ADDR} ^38\.

-- to a special page for years and years. The last time, the only time a real person touched base via the provided graphic e-mail address was on March 27, 2009.

(I should probably save server resources and just use: deny from 38.0.0.0/8)

Staffa

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4358078 posted 11:36 pm on Sep 2, 2011 (gmt 0)

I have blocked the whole 38. range from 0 to 255 for years and have not missed anything important because of it.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4358078 posted 12:43 am on Sep 3, 2011 (gmt 0)

Thanks - that's what I was hoping to hear :)

As I remember, the last time I considered nixing Cogentco, I changed my mind because of one or two mid-level bots I believed to be beneficial; possibly Speedy Spider (Entireweb) however the road has become narrower and what is beneficial to me now is much more exclusive.

caribguy

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4358078 posted 1:06 am on Sep 3, 2011 (gmt 0)

There are some real users in the 38.100-38.117 range (roughly). Mostly NYC offices (/24 and smaller blocks). Will take a look while at the office tomorrow and post what I have.

caribguy

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4358078 posted 12:38 pm on Sep 3, 2011 (gmt 0)

FWIW, here's my 38. It may be outdated and is certainly not comprehensive. Use at your own risk, do not simply copy & paste...


### 038 ################
# 38.0.0.0/8 PSI except 38.96-109 38.111-119 38.220
RewriteCond %{REMOTE_ADDR} ^(38\.([0-9]|[1-8][0-9]|9[0-5]|110|1[2-9][0-9]|2[013-5][0-9]|22[1-9])\.) [OR]
# 38.96 except 38.96.143 (Martha Stewart)
RewriteCond %{REMOTE_ADDR} ^(38\.96\.([0-9]|[1-9][0-9]|1[0-35-9][0-9]|14[0-24-9]|2[0-5][0-9])\.) [OR]
# 38.97 except 38.97.84 | .99 ( .84 Gordon Bros | Frontier Mgmt etc | .99 various) | .106 | .124 (various Boston)
RewriteCond %{REMOTE_ADDR} ^(38\.97\.([0-9]|[1-7][0-9]|8[0-35-9]|9[0-8]|10[0-57-9]|1[13-9][0-9]|12[0-35-9]|2[0-5][0-9])\.) [OR]
# 38.98 except 38.98.75 38.98.85 38.98.9[67] | 38.98.113 (Dune Capital Mgt) | .118 | .176 | .178 | .181 | Risk mgm assn .194/24
RewriteCond %{REMOTE_ADDR} ^(38\.98\.([0-9]|[1-6][0-9]|7[0-46-9][0-9]|8[0-46-9]|9[0-89]|1[02-68][0-9]|11[0-24-79]|17[0-579]|18[02-9][0-9]|19[0-35-9]|2[0-
5][0-9])\.) [OR]
# 38.99 except 38.99.9[6-9] Blekko / Scoutjet 38.99.128 (various, incl Acuity and BusinessWire) | .130 | .151 (keep watching this for abuse) | .162
RewriteCond %{REMOTE_ADDR} ^(38\.99\.([0-9]|[1-8][0-9]|9[0-5]|1[0147-9][0-9]|12[0-79]|13[1-9]|15[02-9]|16[013-9]|2[0-5][0-9])\.) [OR]
# 38.100 except 38.100.53 Various Wash. DC | .184.0/21 Radius Broadband
RewriteCond %{REMOTE_ADDR} ^(38\.100\.([0-9]|[0-46-9][0-9]|5[0-24-9]|1[0-7][0-9]|18[0-3]|19[2-9]|2[0-5][0-9])) [OR]
# 38.101 except 38.101.20 (38.101.20.0/24 Tampa General Hospital | .178 (Various) | .219 | .220 | .227
RewriteCond %{REMOTE_ADDR} ^(38\.101\.([0-9]|[13-9][0-9]|1[0-689][0-9]|17[0-79]|2[3-5][0-9]|20[0-9]|21[0-8]|22[1-689])\.) [OR]
# 38.102 except 38.102.192 (Various) | 38.102.198 (Revlon)
RewriteCond %{REMOTE_ADDR} ^(38\.102\.([0-9]|[1-9][0-9]|1[0-8][0-9]|19[13-79]|2[0-5][0-9])\.) [OR]
# 38.103 except 38.103.63 (Crystal cruises)
RewriteCond %{REMOTE_ADDR} ^(38\.103\.([0-9]|[1-57-9][0-9]|1[0-9][0-9]|2[0-5][0-9])\.) [OR]
# 38.104 except 38.104.69 |.166 | .167 (various NJ/NYC)
RewriteCond %{REMOTE_ADDR} ^(38\.104\.([0-9]|[1-57-9][0-9]|6[0-8]|1[0-57-9][0-9]|16[0-589]|2[0-5][0-9])\.) [OR]
# 38.105 except .67 | .146 (Food Network / Compagnie Financiere de CIC) | (.67.128/26 Washingtonian Magazine | .100 | .150 Cerberus Capital | .170 James Ca
ird | .233 various)
RewriteCond %{REMOTE_ADDR} ^(38\.105\.([0-9]|[1-57-9][0-9]|6[0-689]|1[1-689][0-9]|10[1-9]|14[0-57-9]|15[1-9]|17[1-9]|2[0-245][0-9]|23[0-24-9])\.) [OR]
# 38.106 except .206
RewriteCond %{REMOTE_ADDR} ^(38\.106\.([0-9]|[1-9][0-9]|2[1-5][0-9]|20[0-57-9])\.) [OR]
# 38.107 except 38.107.203/24 Leading Edge Communications (Wireless Provider TX)
RewriteCond %{REMOTE_ADDR} ^(38\.107\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[1-5][0-9]|20[0-24-9])\.) [OR]
# 38.108 except .205 | 38.108.208.0/21 Natural Wireless |.249 Polaris Management
RewriteCond %{REMOTE_ADDR} ^(38\.108\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[235][0-9]|20[0-467]|21[6-9]|24[0-8])\.) [OR]
# 38.109 except 38.109.32/22 Kansas Broadband Internet | 38.109.75 (various) | 38.109.88-91 Natural Wireless | .106
RewriteCond %{REMOTE_ADDR} ^(38\.109\.([0-9]|[124-6][0-9]|3[017-9]|7[0-46-9]|8[0-7]|9[3-9]|1[1-9][0-9]|10[0-57-9]|2[0-5][0-9])\.) [OR]
# 38.111 except 38.111.118.0/23
RewriteCond %{REMOTE_ADDR} ^(38\.111\.([0-9]|[1-9][0-9]|1[02-9][0-9]|11[0-7]|2[0-5][0-9])\.) [OR]
# 38.112 except 38.112.12 |38.112.28 (various) | .81.96/27 Oglivy Renault Montreal | .100 | .120.160/27 Xela | .155 various | .183 | .225 Various small or
gs in Seattle
RewriteCond %{REMOTE_ADDR} ^(38\.112\.([0-9]|[3-79][0-9]|1[01346-9]|2[0-79][0-9]|8[02-9]|1[13-79][0-9]|10[1-9]|18[0-24-9]12[1-9]|15[0-46-9]|2[013-5][0-9]
|22[0-46-9])\.) [OR]
# 38.113 except 38.113.27 Credit union on-line | Brookfield financial | various
RewriteCond %{REMOTE_ADDR} ^(38\.113\.([0-9]|[13-9][0-9]|2[0-689]|1[0-9][0-9]|2[0-5][0-9])\.) [OR]
# 38.114 except 38.114.64-71 .76-79 Wisper Internet IL | 38.114.145 (ThinkEquity Partners) | 38.114.194 (Var. Houston)
RewriteCond %{REMOTE_ADDR} ^(38\.114\.([0-9]|[1-589][0-9]|6[0-3]|7[2-5]|1[0-35-8][0-9]|14[0-46-9]|19[0-35-9]|2[0-5][0-9])\.) [OR]
# 38.115 except 38.115.5 (/24 Resource America) | .17 | .144 | 38.115.155 ( /24 MAN Financial)
RewriteCond %{REMOTE_ADDR} ^(38\.115\.([0-46-9]|[2-9][0-9]|1[0-689]|1[0-36-9][0-9]|14[0-35-9]|15[0-46-9]|2[0-5][0-9])\.) [OR]
# 38.116 except 38.116.192 (Toronto District School Board) | .193 (Various) | 38.116.20[0-3] (38.116.200.0/22 Peel District School Board) | 38.116.36 (see
next line)
RewriteCond %{REMOTE_ADDR} ^(38\.116\.([0-9]|[124-9][0-9]|3[0-57-9]|1[0-8][0-9]|19[14-9]|2[1-5][0-9]|20[4-9])\.) [OR]
# 38.116 except 38.116.36.2nn Alston & Bird LLP Atlanta
RewriteCond %{REMOTE_ADDR} ^(38\.116\.36\.([0-9]|[1-9][0-9]|1[0-9][0-9]))$ [OR]
# 38.117 except .139 | 38.117.151 | .172 | .176 Boston Properties | .182 | 38.117.203 (/24 Sard Verbinnen)
RewriteCond %{REMOTE_ADDR} ^(38\.117\.([0-9]|[1-9][0-9]|1[0-2469][0-9]|13[0-8]|15[0-689]|17[013-57-9]|18[013-9]|2[1-5][0-9]|20[0-24-9])\.) [OR]
# 38.118 except LiteCast / Balticore and Marymount University
RewriteCond %{REMOTE_ADDR} ^(38\.118\.([0-9]|[1-47-9][0-9]|5[01]|6[4-9]|1[0-9][0-9]|2[0-5][0-9])\.) [OR]
# 38.119 except 38.119.128 | .129 | .132
RewriteCond %{REMOTE_ADDR} ^(38\.119\.([0-9]|[1-9][0-9]|1[014-9][0-9]|12[0-7]|13[013-9]|2[0-5][0-9])\.) [OR]
# 38.220 except Westbrook Partners
RewriteCond %{REMOTE_ADDR} ^(38\.220\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-245][0-9]|23[0-8])\.) [OR]
#

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4358078 posted 7:01 pm on Sep 3, 2011 (gmt 0)

Thanks caribguy, that helps.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved