londrum - 1:52 pm on Jul 31, 2013 (gmt 0)
instead of comparing today's file to the database, maybe you could use php to compare it to yesterday's file instead. just search for each product name (or whatever) and if the line is different stick it in an 'update script'. if its new then stick it in an 'import script'. then get php to run both scripts. that will reduce the load on your database by loads, i would imagine.
it will probably still take php a long time to compare the files (considering that it will have to loop through 300,000 lines 300,000 times, but you might be able to knock a load of hours off.
i would imagine that the line numbers for each product are roughly similar from day-to-day (give or take a thousand lines, or whatever), so you should start the script looking at a particular line, based on what it's current line is. and stop it looking one it's done a thousand lines (or whatever). that way you can reduce the time even more -- you'll only have to loop through 300,000 lines 1000 times. in fact, that might be a good way to reduce the time taken when looking through the database as well, if you decide to stick with that way. don't let it look through the entire database, just the 1000 lines each side of its current line number.