Forum Moderators: DixonJones
1) What log traces googlebot activity (I have Apache http)?
2) What tool can summarize my log data and give me info about how deep googlebot goes?
Thank you Thank you
300-500mb isn't a lot to search; when the 'new Googlebot?' thread started on Saturday I searched through 13GB on one of our servers to see how active it was.
If you're not comfortable importing the results into a database for analysis, you can analyse the logs with any competant Web stats package. Often, you'll find that you can run the stat's software with a parameter to include only Googlebot referrers so you don't even need to split them off yourself.