Welcome to WebmasterWorld Guest from 54.159.190.106

Forum Moderators: open

Message Too Old, No Replies

How to merge two tables efficiently in MySQL

   
6:29 pm on Jul 19, 2009 (gmt 0)

10+ Year Member



Folks,

I need your help in figuring out how to merge two tables -

1) Both table columns are identical.
2) First table is the master production table
3) Second table is the temp table that sucks data from various feeds nightly.

What I want to do -
1) Lookup if row is already there in master table (determined by product SKU - already indexed),

1 a) If there, update data e.g. availability column, price column.
1 b) If not there, insert record into master table.

2) If record is not in temp table and is there is master table, mark the status column as 'inactive'

I have written a program that loops over each row of temp table and does the above, problem is that it takes 1 hr every 10,000 rows and I have ~ 1Million.

Is there a more efficient way to do this?

Please help.

thx.

4:41 am on Jul 21, 2009 (gmt 0)

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member



This does the first part (insert missing from master, and update if exists):
INSERT INTO `master` SELECT * FROM `temp` ON DUPLICATE KEY UPDATE `master`.`field1` = `temp`.`field1`, `master`.`field2` = `temp`.`field2`

Secondly do something like:

UPDATE `master` SET `status` = 'inactive' WHERE `sku` NOT IN (SELECT `sku` FROM `temp`) 

Both are untested but should give you a good general direction to doing this without having to create a massive loop.

6:39 pm on Jul 23, 2009 (gmt 0)

10+ Year Member



Thanks, I will try this and post performance gains.

I did add indexes on multiple items and that helped speed up the lookup operation and reduced 1 hr every 10K rows to more like 10 min.