ips-agent

[keyplyr]

UA: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1
Protocol: HTTP/1.1
Robots.txt: Yes
Host: verisign.com
69.58.176.0 - 69.58.191.255
69.58.176.0/20

[lucy24]

That one sounded familiar. Do yours show up in isolation? Mine always come in threes:

69.58.178.dd - - [01/Jan/2016:22:27:45 -0800] "GET /robots.txt HTTP/1.1" 301 545 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1" 
69.58.178.dd - - [01/Jan/2016:22:27:49 -0800] "GET / HTTP/1.1" 403 1763 "-" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1" 
69.58.178.dd - - [01/Jan/2016:22:28:05 -0800] "GET / HTTP/1.1" 403 1763 "-" "BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent" 

Looks like about once a month; I just grabbed one at random. And wtf? What were they doing, requesting material on 26 December 2013 from a site that wasn't even in the index yet? (I checked: the Googlebot did its full crawl on the 24th; human traffic started trickling in on the 28th.) If it's coincidence, it's a slightly creepy one.

See that 301 for robots.txt? They have never actually seen robots.txt on this site, because all requests, ever, met with a 301 which they didn't follow. Headers confirm that they ask for the with-www form everywhere, so it's 200 or 301 depending on site. Looks like I started blocking them in early 2013 --originally by IP, now by header. Before then, they did some second-order crawls (example.com/directory/ though rarely any further) but never asked for material in a roboted-out directory. I suspect this is coincidence.

Oh, fancy that. In logged headers:

From: [e-mail]

[jmccormac]

I think it is Verisign doing usage and development measurements in their TLDs.

Regards...jmcc

[keyplyr]

"I hear the distant whistle and feel the vibration of the tracks, I see that the bridge trestle is broken, but I have no idea what to do about it."
- jdMorgan

[jmccormac]

There is a bit of a tumbleweed vibe about WW of late.

Regards...jmcc

[not2easy]

They were first blocked on my sites in 2013 for visiting resources blocked in robots.txt - using the same agents listed above and in threes. Informational sites, same 3 visit/2 UA patterns.

[keyplyr]

I can recall paying them $160 per year to register a TLD when they had the monopoly in 1998 & 1999. The next year it was $80 when US congress further expanded ICANN role. Then of course it became competitive after that... still I've always held a grudge :)

[jmccormac]

If you were paying $160, you were being ripped off. It was $100 for two years for.COM (after 1995). It dropped to $75 for two years and some who had paid the full amount got a refund cheque. There was a time when it was free to register a .COM.

Regards...jmcc

[keyplyr]

Maybe in Ireland, not where I was. I registered about a dozen in '98... but I agree that I (and everyone else) was "ripped off" by Verisign. I also remember they required a fee to move the assignments (can't remember how much.) I sold a few and had to jump through hoops.

Anyway, so much for nostalgia :)

[jmccormac]

They must have had a special price for you. :) The $100 and later $75, were the official prices charged by Network Solutions. It was the registry and only registrar. NetSol used to send out paper invoices by post back then. If you registered via your ISP/HSP, then they probably added their own charge.

Vaguely back on topic, ips-agent is probably a lot more benign than some of the bots seen. From reading the Verisign domain publications, it may be used to publish the figures of single page sites and sites with more than one page in those documents. Measuring development and usage in a TLD is a very difficult task. (Speaking from experience.)

Regards...jmcc

[lucy24]

usage and development measurements in their TLDs.

Hard to see how-- assuming you really just mean, er, D-- since a couple of the visited domain names never existed before me, afaik. (We're not talking about plausible-set-of-keywords dot com.) Or are we talking about all domain names, ever?

:: detour to check ::

Oh, uh, yeah, I guess "\d+ .+" would be more likely to yield results if I enabled Grep.

Closer investigation tells me they never worked from a shopping list. I can tell because requests used to be for /directory/index.html, later changing to /directory/ alone with no intervening redirect. (Based on logs, I got rid of internal "index.html" links in October of 2011. And high time too. Still higher, unfortunately, by the time I instituted a comprehensive redirect almost a year later. Bleahh.) That explains why the 403d requests were never for anything but the front page: they don't remember earlier visits.

[jmccormac]

The typical approach is to check the websites (if they exist) for all active domains in a TLD's zone file. The percentage of domains having websites can be anything from 60% to 85% and not all websites actually respond. This doesn't work on Google's infinite monkeys approach of following links. Sometimes, with a TLD as large as .COM, a smaller set of random domain names is chosen and these are surveyed. The .MOBI registry also used to do a regular survey for mobile ready websites and did cover TLDs outside .MOBI. However that TLD has declined.

Regards...jmcc