cookies missing from a large percentage of records in log file

Large percentage of records in log file does not show cookies

[sebs]

We wanted to use persistent cookies on all our pages. So just to test it out, we put the code for cookie in just one page. We also have enabled asp session tracking. We imported one day’s log file into a SQL database. The results have been surprising.

Page views: 1490
Records with no cookie: 185
Records with cookie inserted by us: 185
Records with ASP session id in the cookie: 735
Records with siteserver=id=a-long-number: 5

Here are specific questions
a.We never did anything specific to get siteserver=id in our log file. We have no idea how it got there. And that too on just 5 records out of 1490. How did this happen?
b.We have enabled asp session id. What I don’t understand is that there are 735 records with asp session id in the cookie, but only 185 records with our variables in the cookie.

I assume that part of the traffic would be crawlers and people who disable even first party cookies. But would that number so high?

Would appreciate some hints on where to look for a solution?

The whole idea of persistent cookies was to do better tracking

[cgrantski]

All these stats have to do with only one page file's records? Not the whole log, right? There were 1490 requests for that pagefile?

[sebs]

That's correct. 1490 is the page visits for just that page. All the numbers we got from running queries on the database.

[cgrantski]

If it were me I would start with the fact that when a cookie is issued by a page it may not actually show up in the hit for that page. I don't have a lot of experience with sites that issue cookies sporadically like this, but I would take that day's logs, throw away lines not having to do with page files, and sort it with Access or Excel into sessions (based on a concatenation of IP and UA field plus time). I'd examine each of those visits that contained your cookied page, and see what's happening in the cookie field. You might find that your cookie is happening but not showing up until the next page view's log line. If it is, that's your answer. If on the other hand it's tending to not show up at all, while the ASPSESSION cookie is, I'd start thinking about your custom cookie's programming.

[Ben_Graham]

The first request for a page that adds a cookie WILL NOT include the cookie. Think of it this way:

A user requests a page. Your server records the request in the log. The response includes a command to save a cookie in with the HTML, etc.

For the second request (usually an image on the page) the request includes the cookie (all requests include the cookie value when it exists). This cookie is dutifully recorded in the log along with all the usual data.

[sebs]

cgrantski and Ben, thanks so much for your inputs.

No, I didn't realize the bit about first request not including the cookie. I think both your inputs combined answers my question to some extent. Testing cookie for just one page probably was not a good idea. We will do it on a few more pages and then collect the results again. I will post the results here again

[sebs]

One more thing I can try. Look for all records that have this cookie page as a referrer. By this time, the records should have cookies. Right? I will do that and post the info here.

[Ben_Graham]

You can also try checking images on that page, which usually won't be requested until after the cookie has been set. If you want to be double-sure, you can insert javascript to request an image well after the page is loaded. The Machiavellian would request a 1 * 1 gif every ten seconds, whith a unique URL parameter on the gif, so that you can work out how long they were on the page as well. It would require custom analysis, though.

[sebs]

Just wanted to update based on what i learned from this thread. I decided to analyse pages that come after the page from where the cookie was inserted.

Page A: the page where we were setting cookie
Page B: pages that were referred by page A

I looked at all records for Page B and these are the results
Total pages:847
Records that have the values we explicitly inserted:610
Records that have the session id in the cookie, but not the values we inserted: 237

The good news is that around 72% visits are showing cookies correctly. What we are not able to find is why certain records have asp sessionids showing up in the [cs(cookie)] field in the log file, but not the values we are inserting in the cookie. I tried looking at client ips to see if there is one IP that is skewing the data. That's not the case. They are different IPs.

Hope I'm not confusing anyone here.