Forum Moderators: DixonJones

Message Too Old, No Replies

Tracking Google spiders and crawlers withough log access?

is it possible?

         

foy

7:00 pm on Oct 30, 2002 (gmt 0)

10+ Year Member



I have a site on a virtual server where I have php and cgi features but no logfiles (the admin said they didn't enable them because they would be too big and eat too much space).

So how can I extract referrer data without having access to raw logs? Any ideas? I tried some counter scripts but they were useless since they only track the keyword serps but they do not make transparent if a spider crawled my site.

jeremy goodrich

7:04 pm on Oct 30, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you have php or perl, it should be easy enough to set up a counter which reads the environment variables.

I'd change hosts, personally, but the variables you want to look for are REMOTE_ADDR and HTTP_USER_AGENT for spiders etc. also, for search engine referrals the variable HTTP_REFERER is golden :)

jatar_k

7:19 pm on Oct 30, 2002 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



as jeremy says, you can write your own custom logging. You could use db or flatfiles and use the variables that jeremy mentioned. I have created them before and they work very well.

WebRankInfo

4:57 pm on Nov 5, 2002 (gmt 0)

10+ Year Member



If you don't want to make it by yourself, try my PHP free application "GoogleStats" which does it for you :-)