homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum


 10:13 pm on Feb 9, 2014 (gmt 0)

Has anyone got the remotest idea who these people are?

IP: 38.99.82.abc
They appear to own the whole /24, though all crawling to date is from the narrower .192/26 subsector.

UA: Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)
But see below.

Far as I can make out they belong to an outfit called Fieldtech. The one in Idaho, not to be confused with the (probably) several dozen other companies with the same name. They ask for and appear to obey robots.txt, and generally crawl at a leisurely pace. ("Appear to" = they've only gone two steps from the front page.) I've met them twice: once in the first half of January, and again a few days ago.

Mystifying quirk:
Alongside their ordinary crawling, each visit has included a block of exactly 18 (eighteen) requests for the same interior page. For these they used a different UA
which got them blocked. Eighteen times in rapid succession.



 3:14 pm on Feb 10, 2014 (gmt 0)

Any PSI/Cogent CIDR is unwelcome on our sites.
Currently have 6 ranges blocked, but 38.99. hasn't dared knock on our doors yet...


 9:30 pm on Feb 10, 2014 (gmt 0)

I have the range - blocked as "servers".


 1:33 am on Feb 11, 2014 (gmt 0)

I have the range - blocked as "servers".

I could never do that; I see too many humans from this area. They tend to be Canadian schools,* probably sub-sub-subletting their /27 and /30 slivers.

* I'm pretty sure it was in this area that I first encountered "Separate" as a noun.


 4:49 pm on Feb 12, 2014 (gmt 0)

Dstiles, I have a boatload of humans coming through there, especially 38.100. and 38.108.


 9:49 pm on Feb 12, 2014 (gmt 0)

I need the sub-ranges identified. The current identity (via arin) is useless. Small sub-ranges of /29 or /25 suggest businesses rather than individuals, and a lot of businesses either do not use our services or abuse them (cf comcast).

Also bear in mind that I am in the UK and get so many bad hits from US that I'd really like to block the whole country; pity about my customers, who won't let me do that. :(

If anyone has a reasonable breakdown of the range I will consider opening up some of it, but the holes will need to be at least /20.


 1:56 am on Feb 13, 2014 (gmt 0)

PSI/Cognet is assigned the entire Class A.


 3:52 pm on Feb 13, 2014 (gmt 0)

But so are several other providers. Most of them give breakdows via DNS assignements. Cogent seem too lazy to do this - or they consider the whole /11 to be one big mish-mash of business/server/whatever, which drops them into a hack blacklist anyway.


 10:58 pm on Feb 13, 2014 (gmt 0)

especially 38.100. and 38.108.

Heh. I think at one time I had 38.100 blocked until I realized there are humans in there. Most of mine seem to be from 104 and 107-- but the slivers are so tiny, this really is random.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved