homepage Welcome to WebmasterWorld Guest from 23.20.91.134
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 103 message thread spans 4 pages: < < 103 ( 1 2 3 [4]     
Server Farms - May 2013
Ongoing Hosting Data Center Discussion
incrediBILL




msg:4570885
 8:14 pm on May 5, 2013 (gmt 0)

Continuation of the March 2013 thread:
[webmasterworld.com...]

 

keyplyr




msg:4600157
 11:10 pm on Aug 7, 2013 (gmt 0)

Nice

not2easy




msg:4600979
 11:37 pm on Aug 10, 2013 (gmt 0)

Maybe these are older than my records, but I bumped into two new (for me, anyway) Google ranges this week:
74.125.0.0 - 74.125.255.255 74.125.0.0/16
looking for no-indexed images with User_agent "GoogleProducer; (+http://goo.gl/7y4SX)"

72.14.192.0 - 72.14.255.255 72.14.192.0/18
looking for "HEAD /feed/ HTTP/1.1" with the same UA as the other one.

They visited just 3 minutes apart.

not2easy




msg:4601097
 5:19 pm on Aug 11, 2013 (gmt 0)

Too late to edit, but I finally found out that the two Google ranges have to do with mobile content and rss feeds. I should have tried it earlier, but after finding no info searching for the UA GoogleProducer, I visited the shortened URL in the string and it has an explanation. Maybe this info should be moved to another thread.

dstiles




msg:4601114
 7:07 pm on Aug 11, 2013 (gmt 0)

I blocked 74.125.0.0/16 in Jan 2010. It's been used for really nasty things from G itself in the past.

For what it's worth, I block ALL of the following G ranges with the exception of true googlebot sub-ranges within them (ie there is an rDNS entry for them).

8.8.4.0 - 8.8.4.255
8.8.8.0 - 8.8.8.255
64.233.160.0 - 64.233.191.255
66.102.0.0 - 66.102.15.255
66.249.64.0 - 66.249.95.255
70.32.128.0 - 70.32.159.255
72.14.192.0 - 72.14.255.255
74.125.0.0 - 74.125.255.255
108.177.0.0 - 108.177.127.255
172.217.0.0 - 172.217.255.255
173.194.0.0 - 173.194.255.255
203.208.32.0 - 203.208.63.255
209.85.128.0 - 209.85.255.255
216.239.32.0 - 216.239.63.255

I accept there are some user proxies within the above ranges (translate, rss etc) but they are more trouble than they are worth because most seem to be used for a wide variety of purposes and there is no single usage nor solid rDNS for any of them. Which is typical of G. :(

not2easy




msg:4601310
 3:51 pm on Aug 12, 2013 (gmt 0)

Thanks for that info, I did block the ranges because on the explanation page linked to in the UA it says that the person using the GoogleProducer UA has verified that they own the content requested, which I know is a lie. I had only been blocking the Translate range, so I will look into this list also.

keyplyr




msg:4601339
 5:59 pm on Aug 12, 2013 (gmt 0)



[webmasterworld.com...]

dstiles




msg:4602215
 8:45 pm on Aug 15, 2013 (gmt 0)

Update on rackspace as of June...

192.237.128.0192.237.255.255

Includes cloud.

lucy24




msg:4605264
 8:41 pm on Aug 27, 2013 (gmt 0)

172.255 Ubiquity Servers

Not a brand-new range, but Forums search is silent, so worth repeating.

I checked backward: 172.254 is Time Warner, so just the /16

Edit:
185.5.216.0/22 Poland

Isolated robot from 185.5.219.238. The Polish ranges I've met tend to be so dirty that I've adopted a one-strike rule and don't bother to look up the hostname unless I've previously met humans from the same place.

dstiles




msg:4605521
 7:09 pm on Aug 28, 2013 (gmt 0)

172.255/16 is nobis and bloacked. For nobis I have (all blocked):

23.19.0.0 - 23.19.255.255
64.120.0.0 - 64.120.127.255
69.147.224.0 - 69.147.255.255
70.32.32.0 - 70.32.47.255
108.62.0.0 - 108.62.255.255
108.177.128.0 - 108.177.255.255
142.91.0.0 - 142.91.255.255
142.234.0.0 - 142.234.255.255
147.255.0.0 - 147.255.255.255
172.255.0.0 - 172.255.255.255
173.208.0.0 - 173.208.127.255
173.234.0.0 - 173.234.255.255
174.34.128.0 - 174.34.191.255
216.6.224.0 - 216.6.239.255

I have 185.5.216.0/22 listed as DSL and spot-check of a few IPs for open ports suggests this is so. 185.5.219.238 seems to be a rogue. The range was registered by me in Feb this year and has shown no adverse activity since.

keyplyr




msg:4605619
 2:37 am on Aug 29, 2013 (gmt 0)

Thanks dstiles. I didn't have one of the Nobis/Ubiquity ranges.

wilderness




msg:4607015
 11:46 am on Sep 3, 2013 (gmt 0)

B2 Net Solutions

Two of these ranges are sublets and part of other backbones.

No robots. No supporting files.

B2 Net Solutions
ValueVPS VALUEVPS 198.20.175.0 - 198.20.175.127
198.20.175.42 - - [01/Sep/2013:08:14:28 -0600] "GET /MySub/MySubSub/MyPage.html HTTP/1.1" 403 614 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727 ; .NET CLR 4.0.30319)"
198.20.175.42 - - [01/Sep/2013:08:20:04 -0600] "GET /MySub/MySubSub/SamePage.html HTTP/1.1" 403 644 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

B2NET-173-224-227-0-24 (Juch-Tech Inc (JUCHT)) 173.224.227.0 - 173.224.227.255
B2NET-173-224-236-0-24 (Juch-Tech Inc (JUCHT))173.224.236.0 - 173.224.239.255 173.224.224.0 - 173.224.239.255 173.224.224.0/20
CUST-NETBLK-PHX-173-232-162-0-24-663 173.232.162.0 - 173.232.162.255 173.232.162.0/24
B2-NET-SOLUTIONS 192.157.48.0 - 192.157.63.255 192.157.48.0/20
B2NETSOLUTIONS 192.186.128.0 - 192.186.191.255 192.186.128.0/18
B2NETSOLUTIONS 192.198.96.0 - 192.198.127.255 192.198.96.0/19
CC-192-227-156-8-29 192.227.156.8 - 192.227.156.199 192.227.128.0/17
B2NETSOLUTIONS 192.241.64.0 - 192.241.127.255 192.241.64.0/18
B2NETSOLUTIONS 198.20.160.0 - 198.20.191.255 198.20.160.0/19
B2NETSOLUTIONS 198.154.80.0 - 198.154.95.255 198.154.80.0/20
B2NETSOLUTIONS 198.245.64.0 - 198.245.79.255 198.245.64.0/20
B2NET-208-78-63-0-24 208.78.63.0 - 208.78.63.255 208.78.53.
B2NET1150 (Juch-Tech Inc (JUCHT)) 208.92.115.0 - 208.92.119.255 208.92.112.0/21
B2NETSOLUTIONS 69.4.80.0 - 69.4.95.255 69.4.80.0/20
B2-NET-SOLUTIONS 69.58.0.0 - 69.58.15.255 69.58.0.0/20
B2NETSOLUTIONS 2607:4300:: - 2607:4300:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

dstiles




msg:4607124
 7:32 pm on Sep 3, 2013 (gmt 0)

Thanks. Several there I didn't have. :)

keyplyr




msg:4607334
 8:09 pm on Sep 4, 2013 (gmt 0)



Hmmm... Thanks, I didn't have any of those except the one inside of the Psych range.

This 103 message thread spans 4 pages: < < 103 ( 1 2 3 [4]
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved