Where do we see Googlebot

extreme newbie question

[PFOnline]

Hi, many times I've heard people say "Googlebot is hittin my site right now"

But my question is: Where do we see this info? In logs of some sort? I'd like to be able to see when/if google has visited my site.

I looked for this one in the WebmasterWorld Google FAQ but didn't find it.

[joker197cinque]

You can know this if you can access log files of your web server.

Reading these logs you can characterize two google bots(freshbot 64.xx.xx.xx and deepbot 216.xx.xx.xx)

Hope this helps.

fabriZio

[edited by: joker197cinque at 10:36 pm (utc) on Mar. 10, 2003]

[oilman]

I saw Googlebot laying the ditch last week - looked like he got rolled by and SUV or something trying to cross the information super-highway. <gag - bad bad bad joke> :)

seriously tho - you'll find Googlebot in your log files. You can open up your log files in a text editor if you like and just search for 'goooglebot' or you can use a stats analyzer of some sort and filter for it.

See here: [webmasterworld.com...]

[heini]

>In logs of some sort?
Yes...
Now, what you need is access to your logfiles, as very likely given by your host. Either that or you are given preanalysed logs, as per webalizer or some other standard analyzer.
If in doubt look into your hosts faq, or ask them.

Darn, Oilman was faster and funnier than me.

[PFOnline]

Thanks all.

One more quick question...

Is there such thing like a "log analyzer tool" or something that will better help u read and understand those log files? rather than reading them as raw text?

[heini]

PF, why don't you head over to the Tracking and Logging forum, we have put that online specifically to address the issue of tracking visits , logging them, analyzing them etc. Also there are several threads on what analyzers people use and why.

[oilman]

>>"log analyzer tool"

PFOnline - that thread I listed above is all about log analyzers. Some are free some cost a few $$$. Have a read of that thread and try a few out. They almost all have free trials.

[PFOnline]

Ahh, ok... thanks heini. :D

[PFOnline]

hehe oilman, I actually started that thread you listed above, so I've seen it before. :D

Good idea thanks. :)

[PFOnline]

What i meant was, is there such a thing u can just "plug those raw log files" into, that will like analyze the log files and give u 3D graphs, rather than reading them as raw text, and rather than a stats service where you have to put extra code on your site.

[oilman]

that's how most analyzers work. Lots of them are desktop apps that you just point at the directory on your pc that you've downloaded your raw logs to. They churn away and spit out tables and graphs and such.

[Jesse_Smith]

You can make your own Googlebot log using this code in any script that you use in SSI. Make a google.txt file, set the permission to allow it to be written, and then add this code to the SSI cgi script.

$database = "/path_to_file/web/cgi-bin/google.txt";
$shortdate = `date +"%D %T %Z"`;
chop ($shortdate);

if ($ENV{'HTTP_USER_AGENT'} =~ /googlebot/) {
open (DATABASE,">>$database");
print DATABASE "$ENV{'REMOTE_ADDR'} - $ENV{'HTTP_USER_AGENT'} - $ENV{'SCRIPT_URL'} - $shortdate\n";
close(DATABASE);
}

[TWhalen]

I have a program that allows me to watch my logfiles in real time.
It's pretty neat.

[PFOnline]

TWhalen, that's exactly what I want! Can you tell what the name of the program that is?

[TWhalen]

It's a program my employer custom-built for me in-house.(We may start selling it very soon.)

Basically, I can watch all day long as site visitors come in to our websites and watch (in real time) as they move from page to page. It shows me time, date, client IP address, and referring URL (basic logfile stuff) and allows me the option to instantly trace-route or ping a visitor, or open a new browser to any refferring URL.

I can also add all kinds of custom filters, sort, or search the data in any number of ways.

It doesn't do graphs or anything, but seeing visitor behavior in real time is something that can't be deciphered from a graph anyhow. It's by far the best tool I've ever used for SEO.

Sticky me your email address and I'll let you know when we start offering it to the public.

[Rugles]

PF

I use LiveStats.

It is part of my hosting package. I can see who is on the site up to the minute. Plus see what pages they are looking at, where they came from and a lot more data than you would ever want. Plus you can set warnings and special tracking.
There is another competing program but I forget the name.

These programs are a great tool.