Forum Moderators: DixonJones

Message Too Old, No Replies

Referrer Domain List to file for page inclusion?

         

slobizman

11:00 pm on Jan 22, 2004 (gmt 0)

10+ Year Member



I've searched the web for hours and hours looking for a product or service I need, and cannot find it. I'm hoping someone can help.

The end result of what I need is to have a file created on demand that lists the referral domain names for a period (the last week is my preference, in highest to lowest order of hits). I would then take this file and include it in a PHP page (my blog index page) for the purposes of listing referring sites. I'd run this script to create a new file each week.

I have found many tracking programs that provide lots of data in charts and lists, but have not found any that will give me this simple list.

I do have full logs with referrer info on my Apache server, MySQL and PHP available. (Also have Analog installed).

A program that reads the log files would work, or a program for which I have to place some code into my main pages to log hits (if it can handle tens to hundreds of thousands of hits a day) is okay to. Or, on that subject, even a hosted service would work too, if that's the only option too.

Any ideas?

dcrombie

10:39 am on Jan 23, 2004 (gmt 0)



Here's one I prepared earlier ;)

awk -F[\"] '{print $4}' yourdomain-combined_log \
¦ sed 's/\?.*$/\1/' \
¦ sort ¦ uniq -c ¦ sort \
¦ grep -v yourdomain.com > somefile.txt

This will read an entire log file (or a list of log files) so you might need to do some pre-parsing to focus on one week - or set your logs to rotate weekly.

slobizman

1:42 pm on Jan 23, 2004 (gmt 0)

10+ Year Member



Cool!

However, I'm obviously not as technical as you. Doing these type of commands is my biggest weakness. I'm completely confused looking at it.

Do I put that in a file and run it in my SSH terminal window?

Would you mind giving a few instructions to a complete idiot?

dcrombie

2:05 pm on Jan 23, 2004 (gmt 0)



Type just the first line into your SSH window (without the trailing \) and see what happens. Then type the first and second lines, and so on. Don't copy and paste as this forum mangles some of the characters.

When you get to the end you should find that a file is created containing the information you want.

You'll need to replace yourdomain-combined_log with the path to your log file/s.

You can put the command into a shell script and run it using cron - or call it from PHP using system or exec.

slobizman

9:51 pm on Jan 23, 2004 (gmt 0)

10+ Year Member



Thank you. I'll try that.

Dumb question: can you tell me how to set up a shell script? Is it just a file with a ceratin extension?

dcrombie

12:37 pm on Jan 24, 2004 (gmt 0)



How to write a shell script [vertigo.hsrl.rutgers.edu]

onlineleben

1:33 pm on Jan 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



OK, here is the non-technical appraoch:
copy the referer-stats from your logfile-analysis ( I use analog and that makes it simple, as the output is text on a webpage ) and paste it into an Excel spreadsheet. Everything starting with www is converted into a hyperlink by Excel.
Save the file as webpage and copy/paste the code into your destination file (eg. your blog)

Hope this helps

slobizman

3:54 pm on Jan 24, 2004 (gmt 0)

10+ Year Member



Thanks for the great link!

I also have Analog on my server. My problem is that I want to look at the statistics by week, but I only see that I can see it by day. Is there a way to configure that program to show data by week as well as day that you know of?

Sorry for so many questions.

onlineleben

10:38 pm on Jan 24, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Regarding Analog:
Format is YYMMDD
Below is for one week (have to do the changes in analog.cfg manually.
There probably also is some syntax for a week period (like the yesterday example below, but I have to look it up.

# You can limit the dates analysed with FROM and/or TO: see docs/include.html
FROM 040101
# FROM -00-00-01 [yesterday]
TO 040107

slobizman

3:21 pm on Feb 4, 2004 (gmt 0)

10+ Year Member



onlineleben mentioned doing a manual method. I've started thinking about this and now have my Analog program providing a weekly list of referrer, which I can copy into Excel or Word.

But there is one more step I need to be able to do. Some of the "referrers" don't really have a referral link set up for me--someone might have mentioned my site in their discussion forum for example. Some are search engines. What I need is a program that can read through the Word, Excel or ASCII file which contains one domain URL per line, and somehow tell me which sites do not have a link, so I can remove them from the list.

I've looked around and cannot find anything like this. There are plenty of referring sites link checking programs but not anything that does something as simple as I'm trying to do.

Does anyone have any suggestions?