Deduping huge keyword lists - Clean software?

[EvilDan]

Can anyone recommend some software that isn't filled with nasties, that can de-dupe keyword lists please?

[your_store]

You can use a pivot chart in Excel, as long as you're not worried about filtering out permutations, ie it won't filter "blue widgets" and "widgets blue".

[EvilDan]

oooh - that does sorta suck

anything else guys?

[4thePegeh]

google's traffic estimator will remove redundancy

but I think it has a limit on the number of phrases per use

[mikedee]

You can create a database with a unique key on the keyword and then insert your data, any duplicates will not be inserted.

Without sql you could write a quick php script that uses the keywords as the keys of an array, this will stop duplications and you can get the de-duplicated keywords back with array_keys. There are also more specific array functions like array_unique.

There are probably a few pieces of software about for it too, but its a very simple job with a bit of scripting.

[aruns]

Excel 2007 has a tool to remove duplicates. Not sure how it processes blue widgets & widgets blue. I guess you can test it & see if this suits your requirement.

[adwatson]

If you're a coder you can do it yourself with Perl (or PHP) by creating a hash/array, use each keyword as the key, then print out all the keys.