homepage Welcome to WebmasterWorld Guest from 54.227.215.140
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
Forum Library, Charter, Moderators: Receptional & mademetop

Website Analytics - Tracking and Logging Forum

    
Raw logs
gruntre

10+ Year Member



 
Msg#: 2248 posted 7:58 am on Apr 20, 2004 (gmt 0)

I have my 1st hosting account am am trying to learn the ropes.
I have accessed the raw logs and the file is very large containing stuff like this:
999.999.999.9 - - [20/Apr/2004:17:41:20 +1000] "GET / HTTP/1.0" 200 2369 "http://www.google.com.au/search?hl=en&ie=UTF-8&oe=UTF-8&q=video+camera+wide+angle+lenses&meta=cr%3DcountryAU" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
999.999.999.9 - - [20/Apr/2004:17:41:48 +1000] "GET /html/home.htm HTTP/1.0" 200 1789 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
999.999.999.9 - - [20/Apr/2004:17:41:50 +1000] "GET /html/toppframe-new.htm HTTP/1.0" 200 2532 "http://www.example.com/html/home.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
999.999.999.9 - - [20/Apr/2004:17:41:50 +1000] "GET /html/left-frame.htm HTTP/1.0" 200 6776 "http://www.example.com/html/home.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
999.999.999.9 - - [20/Apr/2004:17:41:50 +1000] "GET /intro%20movie/intro%20redeye%20added.swf HTTP/1.0" 200 58562 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)"
999.999.999.9 - - [20/Apr/2004:17:41:53 +1000] "GET /gif/logo%20small%20anim.gif HTTP/1.0" 200 7412 "http://www.example.com/html/toppframe-new.htm" "Mozilla/4.0

Is there a tutorial somewhere I could read to help me understand this and work out how much of a problem it is if at all.

[edited by: tedster at 2:47 am (utc) on April 21, 2004]
[edit reason] remove specifics [/edit]

 

HelenDev

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2248 posted 8:25 am on Apr 20, 2004 (gmt 0)

999.999.999.9 //This is the IP address of the user.

[20/Apr/2004:17:41:48 +1000]//this is the date and time that the page was requested

"GET /html/home.htm HTTP/1.0" 200 1789 //this is the page requested on your site

"http://www.example.com/" //this is the referrer - the page that the user came from

"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" //this is the browser they were using

[edited by: tedster at 2:48 am (utc) on April 21, 2004]
[edit reason] remove specifics [/edit]

gruntre

10+ Year Member



 
Msg#: 2248 posted 9:41 am on Apr 20, 2004 (gmt 0)

Thanks Helen,

Rather tedious to read though I must say.

They have AWstats installed but it doesn't quite give me what I want. I am really interested in unique visitors to my pages each day not hits or revisits.

HelenDev

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2248 posted 10:12 am on Apr 20, 2004 (gmt 0)

There are other stats analyzer programs out there if you do a search. I use the one provided by my web host, which is very basic, to say the least. If I need any more specific info, which I am often asked for, I usually have a trawl through the raw logs.

My raw logs are pretty big too, but I downloaded this little program called Textwiz from somewhere (I forget where), which counts instances of a specific phrase within a text doc, and this is useful for seeing how many times a particular page was a referrer for instance.

Helen.

gruntre

10+ Year Member



 
Msg#: 2248 posted 10:31 am on Apr 20, 2004 (gmt 0)

cool, nice tip thanks.

CYA

ppg

10+ Year Member



 
Msg#: 2248 posted 10:35 am on Apr 20, 2004 (gmt 0)

Hi Gruntre,

You might find this thread on how to track visitors [webmasterworld.com] an interesting read if you havn't seen it already.

cgrantski

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2248 posted 3:40 pm on Apr 20, 2004 (gmt 0)

Have them turn on logging for the query string field if it's not on - Can't really tell from these particular pages.

[edited by: cgrantski at 3:45 pm (utc) on April 20, 2004]

cgrantski

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2248 posted 3:41 pm on Apr 20, 2004 (gmt 0)

And the cookie field, if your site gives out a cookie.

bruhaha

10+ Year Member



 
Msg#: 2248 posted 5:00 pm on Apr 20, 2004 (gmt 0)

Helen gives a good summary of the main points, so this is just to clarify a couple of details:
"GET /html/home.htm HTTP/1.0" 200 1789 //this is the page requested on your site

To be technical, only the "/html. . . htm" part is the actual page/file requested (and its path). The rest of the line provides other details about the visitor's request (and the results):

"GET" is the method ("GET" is by far the most common;but there is also "POST" [often used for form input] and "HEAD" [used by spiders checking links, metainformation...])

"HTTP/1.0" is the version of the HTTP protocol the visitor used to make the request (you should also see plenty of 1.1, its "successor")
"200" is the "server status code". It tells you what sort of response the server made to the user's request.

These can be roughly summarized thus:
1xx -informational (rare! not used at all in HTTP/1.0)
2xx -successful ["200" should be the most common code you see]
3xx -redirect, 4xx -client error, 5xx -server error
(see [helpwithpcs.com ] for a fuller list)

"1789" is the number of bytes [size of the file]the user received (Since a "HEAD" request [and sometimes others] downloads no bytes, it will read - )

"Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" //this is the browser they were using

As you probably can see, this "user agent" part includes not only the browser, but also the operating system of the visitor's machine.

Any decent tracking analysis software will interpret this for you, but it does help to understand it and be able, at least occasionally, to look directly at the logs.

karmov

10+ Year Member



 
Msg#: 2248 posted 7:51 pm on Apr 20, 2004 (gmt 0)

If you're having a hard time getting your stats package to do exactly what you want, don't forget your best friends when it comes to log files; Unix commands!

tail, grep, gawk, wc, uniq, etc...

These friends have answered many many many difficult questions for me :)

gruntre

10+ Year Member



 
Msg#: 2248 posted 12:47 am on Apr 21, 2004 (gmt 0)

Wow,
Thanks heaps guys for all those responses. I am learning so much through this forum!

I have no idea how to get to the unix command line and use the unix friends that Karmov spoke of but am interested, I had to turn away from a script install because of this a few days ago.
Cheers

cgrantski

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2248 posted 1:23 pm on Apr 21, 2004 (gmt 0)

I miss those unix commands - is there an emulator for Windows?

karmov

10+ Year Member



 
Msg#: 2248 posted 4:30 pm on Apr 21, 2004 (gmt 0)

I got one many many years ago for a 486 laptop I inheritted long past its intended life expectancy. Can't remember what the thing was called, but it gave you all your Unix commands on a DOS prompt. I'm sure it won't take long to find if you hunt around for it. It does exist though, though the commands were limited.

ergophobe

WebmasterWorld Administrator ergophobe us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2248 posted 7:38 pm on Apr 22, 2004 (gmt 0)


I miss those unix commands - is there an emulator for Windows?

Well there are variety of solutions from virtual machines that let you run Unix from within Windows (that lets you have the complexity of Unix with the instability of Windows) to Windows ports for many shell commands. I'm pretty sure you can even get emacs.

There's a sourceforge project called unxutils (note the spelling) that has grep, gawk, sed, sh, zsh and more.

Morten Jorgensen (sp?) has an excellent grep tool as well. Try searching for "grep v2.4" or "grep windows".

There are other grep-like tools from the expensive ones that I have not tried (powergrep and the like) to freebies like BKReplacem (odd but very powerful once you get to know it).

Laurel

10+ Year Member



 
Msg#: 2248 posted 2:51 pm on May 9, 2004 (gmt 0)

Oh good thread!
It already answered most all my questions.
I have one left...

I found this string...
(insert IP address here) - - [09/May/2004:08:41:15 -0500] "GET / HTTP/1.0" 200 2572 "-" "Mozilla/4.0 (compatible; MSIE 6.0; AOL 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322)"

Can anyone tell what it was they accessed?
Usually after "GET" it tells me the file they hit, this time nothing....
Also, usually after / HTTP/ it says 1.1"
This time it says 1.0?

What would cause that?

I am just trying to understand, this all too interesting to me! :-)

Any ideas?

py9jmas

10+ Year Member



 
Msg#: 2248 posted 3:07 pm on May 9, 2004 (gmt 0)

(insert IP address here) - - [09/May/2004:08:41:15 -0500] "GET / HTTP/1.0" 200 2572 "-" "Mozilla/4.0 (compatible; MSIE 6.0; AOL 7.0; Windows NT 5.1; FunWebProducts; .NET CLR 1.1.4322)"

Can anyone tell what it was they accessed?
Usually after "GET" it tells me the file they hit, this time nothing....
Also, usually after / HTTP/ it says 1.1"
This time it says 1.0?

"GET / HTTP/1.0" is the request the user-agent sent. Here it is a GET request (as opposed to, say, a POST to script from a form) to /, ie the client requested plain [example.org...] .

The HTTP/1.0 is the client advertising the protocol and version it supports. Most modern web browsers are HTTP/1.1. Most bots are HTTP/1.0. Most proxies are still only HTTP/1.0 (Squid comes to mind). So a modern browser behing a Squid proxy would be advertised as HTTP/1.0. Note the user-agent string includes AOL 7.0 so I assume the user is behing AOL's proxy servers.

Jon.

Laurel

10+ Year Member



 
Msg#: 2248 posted 3:17 pm on May 9, 2004 (gmt 0)

Wow thanks for the QUICK reply!
I think I understand most of what you said. :-)

So what your saying is they hit my front page?
[****xx.com...] ?

Cause my counter doesn't show any hits?

Laurel

10+ Year Member



 
Msg#: 2248 posted 10:44 pm on May 29, 2004 (gmt 0)

OK here's another one...
HEAD /******.jpg HTTP/1.1" 200 0 "-" "-"

usually I see get /***.jpg
what does head mean?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / WebmasterWorld / Website Analytics - Tracking and Logging
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved