FirstGov.gov Spider

[msgraph]

FirstGov.gov Search - POC:[e-mail]

Just grabbed the robots.txt on one site.

[heini]

Coming from a FirstGov IP, not a Fast IP?

[agerhart]

They have been all over one my sites.....over 40 requests.

[msgraph]

AT&T FirstGov Search, c/o GSA

63.240.211.149
8051cr3.wdc1.attens.com

[The Contractor]

I dont get the IP address in my stats but this is what I do get:

FirstGov.gov Search - POC:[e-mail]*
Total hits Files Pageview Bytes sent ¦ Browser
--------------------------------------------------------------------------------------------------
569 568 566 5083662 ¦ FirstGov.gov Search - POC:[e-mail]
--------------------------------------------------------------------------------------------------

What is this for? Are they just randomly crawling sites?

[Key_Master]

Related thread here [webmasterworld.com].

[mbauser2]

I've got 8 stray hits from them since the 8th of this month, all from the same IP address as msgraph.

I see no obvious pattern to FirstGov's requests: 3 of the requests are for robots.txt, it didn't request the server root, the HTML files requested don't link to each other, and it's not limiting itself to ODP-listed URIs.

It's enough to make me curious, but not enough to make me worry.