How can I track visits from Google bot?

I am very new to this!

[26thMay1999]

I have serached and searched and without changing my entire .html site to .php pages I cannot find an easy way to find out if and when the googlebot and freshbot spiders visit the pages of my site. Is this because there is no easy way?

[Brett_Tabke]

If you have a real host, they should provide you with log files...

[peewhy]

26thMay1999 (it's a date then!)

Have you asked your host for logfiles?

Do you have a control panel or something similar?

[26thMay1999]

Right I've found the log files via the control panel.

Is it now a case of trawling through these huge files and matching the IP address with those of the search engines to get dates,times and which pages they crawled?

[John_Caius]

You can do it by hand, but a better approach is to get some webstats software to decipher the log files for you. There are various free and paid options available - the best option is to visit the Tracking and Logging forum, have a look through recent posts there and post a new question if you can't find the answers you need. :)

[menton]

Hi 26thMay1999

I think Webalizer is free and this does an ok job at it. Webtrends is by far the best, but very expensive

menton

[kmtell]

We Urchin for our webstats. I don't love it or hate it. But, when I look at my raw log files (we have a pretty low traffic site), I open them in notepad and conduct a find for "googlebot" and then check the IP attached to it.

[kpaul]

awstats is free and halfway decent w/SE related stuff from logs...

[tabheaven]

[edit]OOPS, I didn't see that you wanted anything for html-pages. This one require php... But I guess that some other can like it...[/edit]

I don't know if this is against board rules...

Anyway, I use this script for tracking googlebot (and other robots).
[googlestats.com...]

Right now they are working on a new version if I understood it correct. (They are french but the script is available in english).

The script presents what pages and when the robot has visited. It also shows the ip and name of the source of the robot.

//Mattias

[why2kit]

I have a version in coldfusion - send me a sticky for the info

[born2drv]

I second AW stats. It basically tells you what robots visited you, when the last one visited (time/date), how much bandwidth they consumed for the month, etc. Very handy if you dont' want to dig through the log files yourself.

If you have access to the log files, there's a program called Windows Grep which is basically a text analyzer, but it will let you search huge log files for text strings like "googlebot" or whatever you want, and it will only return those lines with "googlebot" in it, so you can see every single page that was requested, when, by what IP, etc.