How Can I Tell if Googlebot Has Visited A Site

[amythepoet]

Hi,

Here is my silly question - How can I tell if gogglebot has visited my site?

I also would like to know how to tell if all of my pages are being seen by googlebot.

Thank you

[kaled]

You need to inspect your logs (these are plain text files) and search for google or googlebot or bot. There have been threads talking of a change of user agent name and I'm not certain what Google are using at the moment. When you find a googlebot entry, repeat the search with the correct user agent name.

Kaled.

PS Logs are normally available from your server's control panel.

[ncw164x]

These are the two I am getting at the moment the top one is the old googlebot and the bottom one is the new, still getting a mixture of both bots requesting pages

HTTP/1.0" 200 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html

HTTP/1.0" 200 "-" "Googlebot/2.1 (+http://www.google.com/bot.html

if you can get access to your sites log file, download it to your hard drive and using textpad do a search for either googlebot.com/bot or google.com/bot to see if they are indexing all of your pages, by leaving the reference to bot on the end of the search will eliminate any other google reference in your log file

ncw164x

[amythepoet]

Hi and thank you.

I just looked at my server logs and I see something called Total User agents and then underneath that are lots of listings with Mozilla/4.0

I know that isn't it, but I can't seem to find what you are seeking.

[amythepoet]

Oh, I think I found my log files finally.

Now, I see 7 files listed and they start with the name of my site

I used the search feature and searched for googlebot or googlebot/2.1 and it did not find anything

I think though from looking at my statisics page, that more files are being viewed. I have looked at my referers list and see many more files listed there thiis morning, than in the past.

[ncw164x]

This will be the stats program which you are using not the raw log file, maybe if you are on a shared server your host does not allow access to the log file just to stats for your site

The raw log file is just lines of text as the examples below
66.196.72.17 - - [17/Jul/2004:04:28:01 +0000] "GET /name of file HTTP/1.0" 200 15905 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...]

216.239.46.63 - - [17/Jul/2004:04:28:19 +0000] "GET /name of page requested HTTP/1.0" 200 10039 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

ncw164x

[amythepoet]

Oh, well, I heard from my host and he said the following:

"To determine of a bot such as googlebot has visited your site, you need to know the IP address of the bot. Once you know that, you can look in your stats program (Webalizer, Urchin, etc) for the IP address and determine when the bot has visited."

I told him I don't know the IP address of googlebot and asked further what to do.

I'm stuck

[amythepoet]

Ok, I believe I have the file now. I downloaded it here, but now I can't open it

I'm on a mac and it says to choose an application and I don't know which applicatin to choose.

You said it's a text file , correct?

[sem4u]

It should be a text file unless it is zipped up.

[ncw164x]

See if a program called textpad can be installed on a mac, its very good for opening large files

[amythepoet]

I found it, hurray, thank you so much