FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov - Crawler, Spider, and User Agent ID forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov

1:55 am on Jun 9, 2002 (gmt 0)

IP/Host: 63.240.211.149 (8051cr3.wdc1.attens.com)
User Agent: FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov

Anybody else find it a little creepy to know the US government is crawling sites?

3:42 am on Jun 9, 2002 (gmt 0)

I guess that is Fast's new FirstGov search, building a little off-site content. [fast.no...]

3:59 am on Jun 9, 2002 (gmt 0)

KM,

Just noticed the same on two of my sites today (ate them both up completely), what do they want? ;)

4:04 am on Jun 9, 2002 (gmt 0)

I had it too. It came to read the robots.txt only and scanned no other files.

5:15 am on Jun 9, 2002 (gmt 0)

I thought that was only supposed to scan government sites? Maybe it got lost...