homepage Welcome to WebmasterWorld Guest from 54.242.231.109
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Level 3 Communications
Anyone getting hits from 8.11.2.xx?
Bewenched




msg:4514002
 1:09 am on Oct 31, 2012 (gmt 0)

They seem to come by every evening, but they aren't grabbing pages and I dont have them blocked.... for now.


8.11.2.10

I have the worlds crappiest log tool I'd literally have to parse the log myself to find out the user agent.

 

lucy24




msg:4514067
 6:29 am on Oct 31, 2012 (gmt 0)

I have the worlds crappiest log tool

Uhm... So get a better one :P

I'd literally have to parse the log myself to find out the user agent.

Find in raw logs:
^8\.11\.2\.10\b

grab all hits, pull into a placeholder file.

replace
^[\d.]+ - - \[\S+ -\d\d00\] "GET \S+ HTTP/1\.[01]" \d+ \d+ "[^"]+" "(.+)" ?\n
with
$1\n

There's your UA.

(Haven't personally met your guys. My notes show assorted AV-type things, a crummy free-hosting place that's attractive to hotlinkers, a minor blog site and Yahoo Mobile. All of these are in the 8.12 - 8.28 range. No idea what goes on in the rest of the neighborhood.)

wilderness




msg:4514073
 7:11 am on Oct 31, 2012 (gmt 0)

Level3 is an old thorn.

A few of us have all or nearly all their ranged denied.

The Class A's (4 & 8) are almost elementary, despite being used by many commercial networks.

lucy24




msg:4514081
 8:24 am on Oct 31, 2012 (gmt 0)

8, sure. But most of the 4's I've met are human-- including at least two WebmasterWorld members ;)

Bewenched




msg:4514311
 6:53 pm on Oct 31, 2012 (gmt 0)

Finally found a decent free one, not too shabby either.

Wilderness, could you share the class A's ranges with us.

wilderness




msg:4514324
 7:17 pm on Oct 31, 2012 (gmt 0)

Bewenched,
I opended up some of these ranges in Feb 2012, however previously I had them all denied (there may even be more today):

166.90.0.0 - 166.90.255.25
171.75.0.0 - 171.75.255.255
192.231.42.0 - 192.231.42.255
199.92.0.0 - 199.95.255.255
204.164.0.0 - 204.167.255.255
206.15.0.0 - 206.15.31.255
206.32.0.0 - 206.35.255.255
207.227.0.0 - 207.227.255.255
207.90.128.0 - 207.90.191.255
209.0.0.0 - 209.0.255.255
4.0.0.0 - 4.255.255.255
63.208.0.0 - 63.215.255.255
64.152.0.0 - 64.159.255.255
67.24.0.0 - 67.31.255.255
8.0.0.0 - 8.255.255.255

dstiles




msg:4514398
 10:10 pm on Oct 31, 2012 (gmt 0)

I have the range 8.11.2.0/24 listed as shopwiki, with 8.11.2.9 and 8.11.2.10 as shopwiki bot. This was back in March 2010. Robtex cnet shows 9, 10 as pep2 and pep1 dot shopwiki so probably still valid.

lucy24




msg:4514401
 10:20 pm on Oct 31, 2012 (gmt 0)

Courtesty SEE's block-edit function:

Deny from 4 (isn't that fun? "Deny from 4.0.0.0/8" if the bare form makes you nervous)
Deny from 8 (ditto)
Deny from 63.208.0.0/13
Deny from 64.152.0.0/13
Deny from 67.24.0.0/13
Deny from 166.90 (assuming the final "25" was a typo for "255" since the alternative doesn't bear contemplating)
Deny from 171.75
Deny from 192.231.42
Deny from 199.92
Deny from 204.164
Deny from 206.15.0.0/19
Deny from 206.32.0.0/14
Deny from 207.227
Deny from 207.90.128.0/18
Deny from 209.0

Or, if you prefer,
RewriteCond %{REMOTE_HOST} ^166\.90 [OR]
et cetera

or conversely (with implied AND)
RewriteCond %{REMOTE_HOST} !^166\.90
et cetera

wilderness




msg:4514437
 12:19 am on Nov 1, 2012 (gmt 0)

(assuming the final "25" was a typo for "255" since the alternative doesn't bear contemplating)


I never make typo's ;)

lucy24




msg:4514444
 12:44 am on Nov 1, 2012 (gmt 0)

Me neither:
Courtesty SEE's block-edit function

Ahem.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved