CodilX - 8:37 am on Mar 2, 2011 (gmt 0)
The biggest problem is, that there can't be a check when adding products. The issue is, that someone adds a product list of about 100 000 products. To insure that the information is added quickly, I use MySQL LOAD DATA LOCAL INFILE after file upload, and modifying the import file. It takes too long importing that data and checking everything while doing the import, but when using the load data it takes literally seconds. So now I have to reverse the check for duplicates.