How can you tell if/when your site has been spidered?

[mikko]

How can you tell if Google for instance has spidered my website and what pages it looked at?

[robotsdobetter]

If your web host gives you acces to logs, you can view them and it will tell you. It would look like this:

[25/Aug/2005:23:01:38 -0700] " GET /robots.txt HTTP/1.1" 200 1128 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "www.some--site.com"

[25/Aug/2005:23:15:44 -0700] "GET /robots.txt HTTP/1.0" 200 1128 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)" "www.some--site.com"

There are services and scripts that will track robots for you.