homepage Welcome to WebmasterWorld Guest from 54.167.173.250
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
StanleyWebSpider/1.0
Ankheg




msg:399657
 9:29 pm on Dec 13, 2004 (gmt 0)

Anyone know anything about this user-agent? I've gotten a dozen or so hits in two distinct visits for the same page on one of my sites from it in the last 24 hours, all from a .gov address, concurrent with MSIE 6.0 / Windows NT 5.1 visits. Checked for robots.txt, which is empty, so I can't comment on whether it *obeys* it, but...

sahp4058.sandia.gov - - [13/Dec/2004:20:07:14 +0000] "GET /robots.txt HTTP/1.0" 200 0 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"
sahp4058.sandia.gov - - [13/Dec/2004:20:07:16 +0000] "GET /osama.html HTTP/1.0" 200 7931 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"
sahp4058.sandia.gov - - [13/Dec/2004:20:12:04 +0000] "GET /osama.html HTTP/1.0" 200 7931 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
sahp4058.sandia.gov - - [13/Dec/2004:20:18:12 +0000] "GET /index.html HTTP/1.0" 200 5423 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"

Makes me wonder what they're up to over there at Sandia. :)

 

fiestagirl




msg:399658
 9:45 pm on Dec 14, 2004 (gmt 0)

Well..if the file name is really "osama" then I can guess what they're up to.

bcolflesh




msg:399659
 9:56 pm on Dec 14, 2004 (gmt 0)

From their site:

"Helping Our Nation Secure a Peaceful and Free World Through Technology"

LOL!

Ankheg




msg:399660
 12:34 am on Dec 15, 2004 (gmt 0)


Well, yes, the page really is named osama.html, and it really is, you know, about Bin Laden. I don't know if I'm ready to believe that a web spider (that requests robots.txt!) really is part of our national intelligence programme. The name really intrigues me, as I can't find any reference anywhere to a "Stanley" spider, and if the intelligence community were going to be going a-spidering, I'd expect them to use a commercial product, or spoof Googlebot, or... something a little less attention-getting than a mystery spider.

Though a "unique" user-agent is a first, the pattern of requests does remind me of the behaviour of some cacheing proxies I've seen in the .nipr.mil range, where (surmising) someone behind the proxy requests the page from a site, and the proxy makes repeated requests for the page, presumably in an attempt to see if the content is dynamic or static.

That's just an observation, though. It'll be interesting to see if anyone else observes this "spider" in the wild, and if there's any discernable similarities between pages it's "spidering". :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved