I am looking for an efficient means of merging a number of mysql tables into one database table. They all have the same column names, but I want to ensure no duplicates are added to the merged output. Imagine there is an email field, which is the field I want to test duplication on.
I am not sure whether this can be done with pure mysql, but I will be merging a number of table rows (1000's) at one time, and need it to be as light as possible.
The merging with PHP is ok, but the duplicate check seems to be where the resources use increases.
Thanks Dreamcatcher, good suggestion, except that it appears a "TEXT" type field (the email field) in a mysql db table cannot be set as unique for some reason. It appears only numeric values can? Is this correct? Setting a unique value here would work though? Below is the mysql error:
#1170 - BLOB/TEXT column 'email' used in key specification without a key length