Examining Unusual Log Activity

Trying to find out the source of large number of hits

[mdchachi]

Is there any way to determine the source of a large number of hits?
My personal web site gets an average of 2500 hits a week. One week
in July I got 130,000 hits. Is there any way to determine the source
of the hits? The main handicap I have is that I don't have full http logs
with referrer info.

All my log entries look something like, with no additional information:
24.1.69.241 - - [25/Nov/2001:00:07:43 -0600] "GET /mdc/jpics.html HTTP/1.1" 200 10550

Given that, is there any way to investigate this any further?
Of course I can sample some of the IP addresses and find out where
they are coming from but that still doesn't tell me who referred them.

Thanks,
Mike

[jammy]

i added my own stats script to every page (php) which logs the referer address etc... the stats go into a database which i can look at and play with because it gives me every detail i want (cos i built it).

so what i'm saying is, add a statistic script to your pages (which will add the info to database - even plain text file), as your server doesn't support full http logs...

[gieckboy]

It looks like that IP belongs to the San Francisco Bay Area branch of @Home. Here's the lookup info:

@Home Network (NETBLK-SFBA-TCI-FRMT-1)
425 Broadway
Redwood City, CA 94063
US

Netname: SFBA-TCI-FRMT-1
Netblock: 24.1.64.0 - 24.1.79.255

Coordinator:
Operations, Network (HOME-NOC-ARIN) [e-mail]
[phone]

Record last updated on 09-Apr-1998.
Database last updated on 2-Dec-2001 19:54:36 EDT.

[mdchachi]

Thanks jammy, that's a good idea. Do you do .shtml to do that or is every page dynamically generated?

Thanks gieckboy. Actually I wasn't concerned about that particular IP address, it was just put in as example of the info in my log. I was mainly trying to find out if there's anything I could do to find out the source of a 48-hour span of unsual activity (from variety of ip addresses) that seemed to indicate that my page may have been mentioned in the media or on a web news/portal.

[netcommr]

That log format is what is called the 'common' format.

usually defined in Apache as:
LogFormat "%h %l %u %t \"%r\" %>s %b" common

if you define in your httpd.conf file:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

then in your VirtualHost define:
CustomLog /pathto/access_log combined

and remove the 'common' CustomLog definition in the VirtualHost.

**You don't need to remove the 'common' LogFormat definition.

This will give you the referrer and user agent in your log file.

If you don't have root access, then ask your server administrator for help.

If you don't use Apache, then this will at least give you an idea of what is possible with your web server software, though implementation will be slightly different.

[jammy]

it was on my own personal site i tested the scripts, which essentially are static pages with PHP code within.

i added an include at the top of every page, which runs a script that logs the referer.

even if you have just a plain html site - you could edit your .htaccess file to process HTML as PHP files :) Else, add any variety of pre-built scripts to log the referer. phpbuilder, webdev, webmonkey etc should have free examples you could look at...