Welcome to WebmasterWorld Guest from 54.167.46.29

Forum Moderators: open

Message Too Old, No Replies

Using a SSI to check for Googlebot

I haven't got access to my log file, how do I see if Googlebot spiders me

     
9:15 pm on Mar 4, 2003 (gmt 0)

Junior Member

joined:Dec 4, 2002
posts:80
votes: 0



How can I use a server side include (eg. Apache XSSI, PHP or ASP) to embed or call a script that checks for:
"Googlebot/2.1 (+http://www.googlebot.com/bot.html)" as the USER_AGENT or "crawl*.googlebot.com" or "crawler*.googlebot.com" as the HOST.
9:50 pm on Mar 4, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15755
votes: 0


php

just include a script that looks at $_SERVER['HTTP_USER_AGENT [php.net]']

if it finds your matches it could write it to a text file.

9:52 pm on Mar 4, 2003 (gmt 0)

New User

10+ Year Member

joined:Jan 24, 2003
posts:12
votes: 0


PHP:

Predefined values:
User Agent: $HTTP_USER_AGENT
Remote IP: $REMOTE_ADDR

Remote name (may not work on your server):
$remote_hostname=@gethostbyaddr($REMOTE_ADDR);

10:12 pm on Mar 4, 2003 (gmt 0)

New User

10+ Year Member

joined:May 20, 2002
posts:10
votes: 0


asp:

Request.ServerVariables("HTTP_USER_AGENT")

10:28 pm on Mar 4, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Jan 1, 2003
posts:212
votes: 0


ASP:

User Agent: Request.ServerVariables("HTTP_USER_AGENT")
IP Addy: Request.ServerVariables("REMOTE_ADDR")

weteo is correct, the host name may not be available on your server - many hosting companies turn this "off" because it places an extra load on the server. But if you are looking to use it to identify Googlebot, it is redundant, as you can do that with the User Agent. IP Address, which you did not mention, is also useful, as it will allow you to distinguish between deep crawler and Freshbot!

If you don't wish to track every access to your web pages, but just those from Googlebot, you can also do this -

If InStr(Request.ServerVariables("HTTP_USER_AGENT"),"Googlebot") > 0 then ...

(log code here)

End If