How to detect GoogleBot

I know he's come, but he's not in my logs

[bokesch]

I've heard a lot about googlebot visiting sites, but in my log files all I am able to view is the IP address of my visitor. How am I supposed to know which one Google Bot is?

[gingerbreadman]

These will help.

[webmasterworld.com...]

[webmasterworld.com...]

cheers
Ross

[andreasfriedrich]

You will need to run your logfiles through a tool that does reverse dns lookups to translate the ip addresses to hostnames.

Or lookup the ip addresses for Googlebot and extract only those records matching those addresses.

Andreas

<added>Ross was faster</added>

[Jesse_Smith]

Using Telnet you can make logs that ONLY contain Google.

cat /logs/web.log? grep googlebot > /web/googlebot.log
cat /logs/web.log? grep 216.239.46 > /web/216.239.46.log
cat /logs/web.log? grep 64.68.82 > /web/64.68.82.log
cat /logs/web.log? grep google.com > /web/google-com.log

? should be a line going up and down (just above the return bar to the right), with a space between it and the .log.

1st one, all Googlebot hits.
2nd one, deepcrawl hits.
3rd one, freshbot hits.
4th one, visitor hits from google.com's page.

[GeorgeGG]

cat /logs/web.log? grep 216.239.46 > /web/216.239.46.log
cat /logs/web.log? grep 64.68.82 > /web/64.68.82.log

Shouldn't the 216.239.46 be 216.239.46. add dot
and the 64.68.82 be 64.68.82. add dot
and www.google.com the google.com page
?

And add
64.68.84.
for crawl11.googlebot.com = 64.68.84.31 range
like:
Query : crawl11.googlebot.com
Offical Name = crawl11.googlebot.com
Aliases =
Addresses = 64.68.84.47, 64.68.84.48, 64.68.84.49,
64.68.84.50, 64.68.84.51, 64.68.84.52,
64.68.84.53, 64.68.84.54, 64.68.84.55,
64.68.84.56, 64.68.84.57, 64.68.84.58,
64.68.84.59, 64.68.84.60, 64.68.84.31,
64.68.84.32, 64.68.84.33, 64.68.84.34,
64.68.84.35, 64.68.84.36, 64.68.84.37,
64.68.84.38, 64.68.84.39, 64.68.84.40,
64.68.84.41, 64.68.84.42, 64.68.84.43,
64.68.84.44, 64.68.84.45, 64.68.84.46

groups.google. for referrers from google groups

GeorgeGG