I am looking for an efficient means of merging a number of mysql tables into one database table. They all have the same column names, but I want to ensure no duplicates are added to the merged output. Imagine there is an email field, which is the field I want to test duplication on.
I am not sure whether this can be done with pure mysql, but I will be merging a number of table rows (1000's) at one time, and need it to be as light as possible.
The merging with PHP is ok, but the duplicate check seems to be where the resources use increases.
Thanks in advance.