How to find Googlebot in my stats DB?

I have tried various SQL queries but can't find it.

[rubenski]

Hi all. This problem I have been battling for a long time without success. I believe though that the solution should be rather simple.

In the topic below I already posed my question about finding Googlebot in my stats database. [webmasterworld.com...]

Since that topic next to IP etc, I have also logged the User Agent in my database. I store the user agent by putting $HTTP_USER_AGENT in my db.

I am however still not able to find any evidence of the Googlebot having crawled my site. I am very, very sure that Googlebot has crawled my site for several times since my last topic, bacause al my latest articles are in Google. Somehow the Google UA is not being stored in my db (other certainly are) OR I am not searching correctly.

I have used the following queries:

SELECT * FROM `pageviews` WHERE agent LIKE '%Googlebot%'; --> Empty result

SELECT * FROM pageviews WHERE LEFT( ip, 7 ) = '209.185' OR LEFT( ip, 5 ) = '64.68' OR LEFT(ip, 7 ) = '216.239' --> Not entirely empty, but no Googlebot.

What am I doing wrong? I really don't get this. I have tought myself to write PHP, create relational databases, etc, and this little problem just keeps nagging me. Sometimes I forget about it for a while, but it's almost coming back to me in my dreams now :P

[Netizen]

Perhaps Googlebot isn't visiting your site? Is your site indexed on Google? Have you tried typing

site:www.yourdomain.com

into Google to see what it gives you?

[curlykarl]

Hi

Maybe you may have a problem with your query, maybe not, I personally have no idea.

In my mind an easier way would be to dump your database into Excel, and use the find command (Alt+F) and enter the term 'Googlebot', thats how I do it :)

HTH

Karl

[rubenski]

Netizen: Yes. I am very sure about this. Over 60 of my pages are in Google and ranking very well. Pages have also been updated regularly in the Google cache. So Google is mostcertainly visiting my site.

CurlyKarl: I believe my queries are ok, but I will keep your suggestion in mind. I'll do ANYTHING to find out how I can track Google.

Doesn't this prove the User Agent is actually logged:

23 April 08:52:41
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
23 April 08:50:07
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
23 April 08:49:48
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
23 April 08:49:46
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
23 April 08:49:31
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)

If these users agents are logged, could there be any way that Googlebot is NOT logged? A stated before I use $HTTP_USER_AGENT to log the UA.

[jatar_k]

user agent should be
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

or
"Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)"

[rubenski]

SELECT * FROM `pageviews` WHERE agent LIKE 'Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html)';

and

SELECT * FROM `pageviews` WHERE agent LIKE 'Googlebot/2.1 (+http://www.googlebot.com/bot.html)'

give an empty result too. No errors there, just an empty result. I think I'll make a dump for Excel just to be sure, but I am starting to believe the bot entries are just not there. But still, other UAs are logged without a problem as you can see in my last post. Strange stuff.

[stevenmusumeche]

Try adding some wildcards:

SELECT * FROM pageviews WHERE agent LIKE '%googlebot%'

If that doesn't work, but you are getting other user-agents in the database, you probably have a problem in your INSERT query. Are you properly escaping quotes?