I have a very poor log-analyzer (suggestions, anyone? Free because my boss is ignorant), so I was just checking my logs by hand to figure out whether Google had spidered my (just-)updated index page. Googlebot has been visiting my site quite a lot lately and I'm ecstatic over the results, but I was just curious.
Anyhow, while in Notepad doing a "find next" to locate google searches and bots, I noticed that several times, the bot appeared within fractions of a minute to spider a page a searcher had just clicked through to from the SERPs.
It could've been coincidence-- it was a high-traffic time, and Google was randomly (to me, it seemed) spidering great swathes of my site. Anyhow, I can't remember if this has been remarked upon before.
Details:
Google searcher came to /foo/bar.html and loaded the entire page, images, .CSS, .JS, and all. (08 Sept 16:06:25)
16:07:13, 64.68.82.38 Googlebot/2.1 shows up to get foo/bar.html, then at
16:11:03 bot 64.68.82.18 got robots.txt, then at
16:11:05 64.68.82.18 got fred/barney.html, then at
16:14:12 bot #64.68.82.18 got bar/foo/ HTTP/1.0.
So, it got the specific page the searcher had clicked on, then robots.txt, an unrelated page, and then the first page's entire directory.
I don't know what that means, but I thought I'd point it out. Isn't that interesting? I didn't know Google did that, and still don't have proof.
Googlebot came back repeatedly over the next ten minutes and spidered the first directory, the second directory, and a number of unrelated pages.
It also repeatedly got robots.txt, my .css file, and two different .js files once each.
But, again, this is just one inexperienced webmaster-newbie with Notepad and the Find function aqnd a PR 5 site, so...
BTW, I'm not entirely sure what HTTP/1.0 and HTTP/1.1 actually mean, but it seems that it indicates that the visitor is examining the contents of a directory to see what to get? I'm not sure, so any enlightenment is welcome.
