Merging Data Tables and removing Duplicates

[username]

Hi all,

I am looking for an efficient means of merging a number of mysql tables into one database table. They all have the same column names, but I want to ensure no duplicates are added to the merged output. Imagine there is an email field, which is the field I want to test duplication on.

I am not sure whether this can be done with pure mysql, but I will be merging a number of table rows (1000's) at one time, and need it to be as light as possible.

The merging with PHP is ok, but the duplicate check seems to be where the resources use increases.

Thanks in advance.

[dreamcatcher]

Create a new table and make the e-mail field unique, this will prevent duplicates. Then copy data between tables using IGNORE to skip over any duplicates without errors.

INSERT IGNORE INTO Table2 SELECT FROM Table1;

Think that should work.

dc

[username]

Thanks Dreamcatcher, good suggestion, except that it appears a "TEXT" type field (the email field) in a mysql db table cannot be set as unique for some reason. It appears only numeric values can? Is this correct? Setting a unique value here would work though? Below is the mysql error:



#1170 - BLOB/TEXT column 'email' used in key specification without a key length


Any help would be appreciated.

[dreamcatcher]

Does the e-mail field have to be a text field? I would have thought a varchar(250) would have been plenty. A text field seems overkill.

dc

[username]

Thanks guys, that did it.