Welcome to WebmasterWorld Guest from 54.167.83.224

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

StanleyWebSpider/1.0

     
9:29 pm on Dec 13, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 5, 2003
posts:133
votes: 0


Anyone know anything about this user-agent? I've gotten a dozen or so hits in two distinct visits for the same page on one of my sites from it in the last 24 hours, all from a .gov address, concurrent with MSIE 6.0 / Windows NT 5.1 visits. Checked for robots.txt, which is empty, so I can't comment on whether it *obeys* it, but...

sahp4058.sandia.gov - - [13/Dec/2004:20:07:14 +0000] "GET /robots.txt HTTP/1.0" 200 0 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"
sahp4058.sandia.gov - - [13/Dec/2004:20:07:16 +0000] "GET /osama.html HTTP/1.0" 200 7931 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"
sahp4058.sandia.gov - - [13/Dec/2004:20:12:04 +0000] "GET /osama.html HTTP/1.0" 200 7931 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"
sahp4058.sandia.gov - - [13/Dec/2004:20:18:12 +0000] "GET /index.html HTTP/1.0" 200 5423 "-" "Mozilla/4.0 (compatible; MSIE 6.0; compatible; StanleyWebSpider 1.0) StanleyWebSpider/1.0"

Makes me wonder what they're up to over there at Sandia. :)

9:45 pm on Dec 14, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:May 14, 2002
posts:378
votes: 0


Well..if the file name is really "osama" then I can guess what they're up to.
9:56 pm on Dec 14, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


From their site:

"Helping Our Nation Secure a Peaceful and Free World Through Technology"

LOL!

12:34 am on Dec 15, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 5, 2003
posts:133
votes: 0



Well, yes, the page really is named osama.html, and it really is, you know, about Bin Laden. I don't know if I'm ready to believe that a web spider (that requests robots.txt!) really is part of our national intelligence programme. The name really intrigues me, as I can't find any reference anywhere to a "Stanley" spider, and if the intelligence community were going to be going a-spidering, I'd expect them to use a commercial product, or spoof Googlebot, or... something a little less attention-getting than a mystery spider.

Though a "unique" user-agent is a first, the pattern of requests does remind me of the behaviour of some cacheing proxies I've seen in the .nipr.mil range, where (surmising) someone behind the proxy requests the page from a site, and the proxy makes repeated requests for the page, presumably in an attempt to see if the content is dynamic or static.

That's just an observation, though. It'll be interesting to see if anyone else observes this "spider" in the wild, and if there's any discernable similarities between pages it's "spidering". :)

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members