csv to database

[Sandro87]

Hello,
I'm thinking about how doing a script.
I have a csv file (date,time,data1,data2,data3 etc.) which would contain a log for my weather station updated every 1 minute.
I want to process this csv file every day at midnight to save the log into a mysql database, the file would contain 1440 lines every day and they will be added to the same file continuously.

Processing every line from the beginning each day would be crazy for the server processor (imagine it when I'm in two months log) , what I'd like need help with is: how would organize in theory the script to analyze the file, find what's the last date/time stored in the database and continue to save lines from the very new one excluding the the ones before?

In theory it seems easy for what I can think of. What I would do is create an array (in the first "pass") of the date/time only and then with while loop a new array with lines only with data/time > than the last stored data in the database, this could work BUT do you think that with THOUSANDS or even millions of lines the server memory usage would be too high?
I estimated 76MB of logfile size each year.

Thanks

[londrum]

you could just write the date at the beginning of each new day, on a new line, and get the script to look for that.
it doesn't matter if the date line isn't in the csv format, because the script will only ever be reading the lines under it.

or... even easier... if you know for a fact that the log will grow by an exact number of lines everyday then just get the script to start looking from a set number of lines from the end.

[Sandro87]

I can't do it, the csv is made by a software not made by me. And no, it could happen often that some days the file won't be updated to be processed.
Do you think it's still doable with my method?

[TheMadScientist]

To provide a concise answer and not outline all the ways I see to do what you are asking about, could you please provide the following:

1.) The format of the date col?
IOW: UNIX timestamp or String?

2.) Are the records prepended or appended to the file?
IOW: Are new lines added to the beginning or end of the file?

3.) Are all lines of the file added to the DB table?
IOW: Is row count of the table always going to be the same as the line count of the file, or are there lines which are not entered for some reason?

4.) Have you considered (using php) reading the contents of the file (created by the program you are using), storing the contents to a 'weekly file', 'monthly file', 'yearly file' or some other type of 'collection file', then deleting the entries from the file the program stores the data to? This way you could use the file created by the program you are using as a 'daily file' and by adding it's contents to a 'collective file' and database at the same time, you would keep the size of the file you read and update both the 'collective file' and database with to a minimum... (It seems like you could even 'archive' the file daily, by saving it as a different name (using php) and then saving a blank file with the same name as the file the program you are using stores the data to.)