How to see how many pages Googlebot crawled?

Visits? hits? Relation?

[vitaplease]

I get a Webtrends daily overview from my provider on which I can see the spider visits.

Last day Google was logged as having "34 visits" and "370 hits". The other day it was "24 visits" and only "28 hits". I wonder why there is such a discepancy between the ratio visits/hits of the two days and is there anyway to estimate how many pages therefor have been crawled? Or do I need the raw Webtrends data for that?

[TallTroll]

Sometimes I see Googlebot come by and ask for robots.txt and /, then leave. Then around once a month it scoops damn near everything over about 12 hours.

I don't know Webtrends at all, does it consider images, or embedded files (.swf, .pdf, .doc etc) to be separate entities, and record them as hits?

Probably it's just the difference in crawl behaviour between a "you're still there then" visit, and a "lets have your content" visit

[Alecto]

vitaplease

We use webtrends. I'm not aware of any way to find out which pages Googlebot has visited using webtrends.

If I wish to see which pages Googlebot has grabbed, I download the raw log files, open them in a text file, and use the search function.

[Torben Lundsgaard]

Alecto

All you have to du is create a new profile in WebTrends with a filter, which includes trafic based on browser activity. Enter "googlebot" as identifer. Save the profile and generate the repport. That's it.

[Alecto]

Thx Torben

Actually, our webtrends report is generated from the server, probably the same as vitaplease. We just request a report for a specific date. I'm not sure if I can change any of the configurations, but I doubt it.