I wondered about the cert service angle, but the site it hit doesn't have anything related to a cert, or e-commerce, or https, etc. Will be interesting to see if it becomes a routine visitor, w/ or w/o UA.
wilderness
6:15 pm on Oct 7, 2009 (gmt 0)
Here's a 2003 link [webmasterworld.com] from Jan's old reference file.
dstiles
8:56 pm on Oct 7, 2009 (gmt 0)
Comodo do firewall and AV software - could be a user - but they also have a search site.
A little identity confusion at comodo.com? Yet another UA:
search.comodo.com COMODOspider/Nutch-1.0
robots.txt? Yes
Pfui
3:19 am on Feb 5, 2010 (gmt 0)
This time, the same UA -- but via a 'bare' (no rDNS) IP:
114.255.52.163 COMODOspider/Nutch-1.0
robots.txt? Yes
114.255.52.163 => China Beijing Comodo (114.255.52.160 - 114.255.52.175)
Internet security/firewall/antivirus/SS/whatever company spidering from China? Oh, the irony!
Lain_se
3:32 pm on Feb 14, 2010 (gmt 0)
This bot has hit my server several times looking at SSL certs that are linked to validation scripts that show when the SSL cert expires. I have also been sent several emails about 90-days before my SSL cert expires offering to renew it with them. Once these folks even sent me an email warning me that there was a security issue related to my cert/server configuration and even though the server had been upgraded over a month before that they still claimed it was vulnerable.
Personally I find this type of spamming much like ambulance chasing lawyers and I for one will NEVER use their certs because of it...and now I block them from my server as well and like magic, no more spam from them either..Go figure!
