Forum Moderators: DixonJones

Message Too Old, No Replies

How to trace googlebot?

trace googlebot

         

gesbos1

1:20 pm on Feb 17, 2004 (gmt 0)

10+ Year Member



How can I trace the activity of googlebot on my site? I know that certain logs contain data about googlebot, but I'm not sure which ones, AND, my logs are between 300 mb and 500 mb each so searching would be impossible. 2 questions:

1) What log traces googlebot activity (I have Apache http)?

2) What tool can summarize my log data and give me info about how deep googlebot goes?

Thank you Thank you

ciml

3:16 pm on Feb 17, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I just search the files using something like grep (I'm usually on Linux, but there are search tools for Windows too).

300-500mb isn't a lot to search; when the 'new Googlebot?' thread started on Saturday I searched through 13GB on one of our servers to see how active it was.

If you're not comfortable importing the results into a database for analysis, you can analyse the logs with any competant Web stats package. Often, you'll find that you can run the stat's software with a parameter to include only Googlebot referrers so you don't even need to split them off yourself.

onlineleben

1:09 pm on Feb 18, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Get an analyzing software like Analog. It is free and available in Windows and Linux versions.
Download your logfile and configure the analyzer to just list the visits from googlebot.