eelixduppy - 6:16 pm on Jul 18, 2012 (gmt 0)
Well how often will you be expecting the external sources to change? How are you going to be getting the updates that are made?
When you do this you're going to want to avoid loading all of the data into memory all at once. For example, if you are grabbing data from MySQL as one of your sources, you want to use the unbuffered version of mysql_query to read the data: [php.net...]
You might also want to consider taking an object oriented approach, since this seems to fit that structure nicely.
Personally, I don't particularly like the idea of copying all the data locally before changing it, but then again I guess that depends on a bunch of things. For example, what if for whatever reason you cannot "normalize" a row? Do you ignore it, persist it to disk for later, etc..?
Also, "lots of data" may mean something different to you than to me. I'd say anything in the thousands of rows would be considered a small amount of data. I work in finance, though, so I'm used to working with millions of data items. :)