Issue saving data from access_log

[circuitjump]

Hello everyone,

I'm having a small problem and wanted to see if anyone had any suggestions on how to go about solving this issue.

I'm using PHP to parse my access_log and add the parsed data into a mysql database table. The problem is that my access_log rars up every 24 hours and starts up with a new empty access_log file.

When it's the same access_log it has no problem double checking that it does not put in any duplicate data but when the access log rars up and starts a new one it can't do that anymore because it checks the first entry in the db table with the first entry of the new access_log file and they are not the same and all of a sudden it starts dumping in duplicate data.

I hope I'm making sense with this. But is there anyway for it not to do that? Is there possibly a different route I can take? Any help is appreciated, I've spent a week messing with it and I'm stumped.

Thanks

[FalseDawn]

No, you're not really making sense.
What data are you extracting from the logs, and what is being "duplicated" precisely, and why is this causing a problem?

[ergophobe]

Every request in your access log will have a time stamp. Find out what the latest timestamp is in your DB. Then if the access log entry is greater than that, add it. If it is equal, check the other elements of the request and if different, add it.

This isn't perfect because if someone manages two requests to the same page at the same time from the same IP, you will reject the second one from your log.