homepage Welcome to WebmasterWorld Guest from 50.17.27.205
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
VeriSign
keyplyr




msg:4358365
 7:05 pm on Sep 3, 2011 (gmt 0)

69.58.178.59 "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3"

VeriSign Infrastructure & Operations
69.58.176.0 - 69.58.191.255

rDNS: no
robots.txt: yes

Ate 403s due to a specific attribute in UA

 

Stevietheman




msg:4358389
 8:36 pm on Sep 3, 2011 (gmt 0)

This exact same UA just fell into my robots.txt trap by trying to access something in a disallowed directory around 12:30 Eastern U.S. time.

Stevietheman




msg:4358390
 8:38 pm on Sep 3, 2011 (gmt 0)

It was in the same IP range as well.

Pfui




msg:4358403
 9:16 pm on Sep 3, 2011 (gmt 0)

Aug. 18th, same IP and UA:

69.58.178.59
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
robots.txt? Yes

July 25th, next-door IP and same not-okay UA attribute:

69.58.178.58
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
robots.txt? Yes

FWIW: Multiple prior years, across multiple sites, none with ecommerce or https --

Some bare IPS:

69.36.158.39
69.36.158.66
69.58.178.39
69.58.178.40
69.58.178.58

Some by Host:

crawl3.pub.colo-fo.brn1.verisign.com
crawl31.pub.colo.fo.brn1.verisign.com
ips-crawl2.colo-fo.ilg1.verisign.com
ips-crawl3.colo-fo.ilg1.verisign.com
ips-crawl5.colo-fo.ilg1.verisign.com
ips-crawl8.colo-fo.ilg1.verisign.com

All always using:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7

robots.txt? Yes

Hmm. I always chalked 'em up to a Network Solutions thing, although the hits never coincided with services/renewals either before or after the corporate changes:

"(VeriSign still retains the registry business which had been originally created within Network Solutions prior to VeriSign's acquisition of the company.)" [en.wikipedia.org...]

Actually, I have no clue why they're crawling.

lucy24




msg:4358419
 10:30 pm on Sep 3, 2011 (gmt 0)

Ooh, Verisign, they're the funny ones. They missed last month but generally come by about once a month. And each visit is identical:

IP: so far*, always exactly 69.58.178.59
UA (spaces added by my log-wrangling code): Mozilla/ 5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/ 20090824 Fedora/ 1.0.7-1.1.fc4 Firefox/ 3.5.3 (but wait!)

pages, at intervals of 2-5 seconds:
/robots.txt
/
/ again, this time with UA BlackBerry9000/4.6.0.167 Profile/ MIDP-2.0 Configuration/ CLDC-1.1 VendorID/ 102 ips-agent
/rats/index.html
/paintings/index.html
/fun/index.html
/fun/judy.html
/hovercraft/index.html
/ebooks/index.html
/games/index.html
/rats/mischief.html

Nine pages (out of 150-ish) in two minutes, once a month, puts them squarely in the "no skin off my nose" camp. I looked them up once but forget what it is they claim to do.


* I've only been monitoring raw logs since the end of March, so what happened before then is anyone's guess.

jmccormac




msg:4359523
 3:42 pm on Sep 7, 2011 (gmt 0)

The ips-agent is the spider for the website survey that Verisign runs for the Domain Brief report from what I remember (it gives a rough breakdown of 1 page sites to multipage sites on page 3).

[verisigninc.com...]

Regards...jmcc

Pfui




msg:4375472
 3:15 pm on Oct 17, 2011 (gmt 0)

VeriSign slips "ips-agent" into another UA:

69.58.178.56
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.22; ips-agent) Gecko/20110905 Ubuntu/10.04 (lucid) Firefox/6.0.2

robots.txt? Yes

Project Honey Pot for the IP [projecthoneypot.org...] shows 4,000-plus visits within a year using four UAs to date:

BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3,gzip(gfe),gzip(gfe)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.22; ips-agent) Gecko/20110905 Ubuntu/10.04 (lucid) Firefox/6.0.2

Doubtless there will be more.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved