Batch conversion to UTF-8

Using iconv on a large number of files

[encyclo]

I've got a collection of a few thousand text files containing French text encoded as ISO-8859-1. I need to convert these to UTF-8, but using iconv on each file individually would be far too long. The standard command for iconv is:

iconv -f ISO-8859-1 -t UTF-8 original.php -o newfile.php

I need to do this for every .php file and recursively down the directory structure. Is there an easier way of doing this? Should I use a script, or is there a better tool to do this automatically?

Thanks for any insight! :)

[MattyMoose]

A quick and dirty way to do every file you can find from your current working directory:

find . -exec iconv -f ISO-8859-1 -t UTF-8 {} -o {}.new \;[code]
That kind of works on my system... My iconv doesn't support the "-o" switch.
A quick other one that might work:
[code]
for i in `ls -R *`; do iconv -f ISO-8859-1 -t UTF-8 $i -o $i.new; done

HTH!
MM

[encyclo]

Thanks MattyMoose - one thing is that there will be a mixture of files, including images, PDFs etc., within the subdirectories. Is there any way of targetting just the files with the extension .php? Is it:

for i in `ls -R *[b].php[/b]`; do iconv -f ISO-8859-1 -t UTF-8 $i -o $i.new; done

The above seems not to work recursively, although it worked fine for the current directory...

Also, would it be possible to create a new directory structure but preserve the same file names - otherwise I'm going to have to rename every file?

For my iconv version, I'm using 2.3.2 from Ubuntu 5.04 (ie. straight out of Debian). -o is the output file.
As you can guess I'm out of my depth with this ;)

[encyclo]

For renaming, could I do something like this?:

for i in `ls -R *`; do rename -f ’s/\.new$//’ *.new; done

I'm just copying your syntax here...

[MattyMoose]

Ya, I found that using the for loops with ls -R doesn't work so well.. You'll probably want to try the "find" statement for that. BTW, I usually copy the files to a test directory first so I don't wreck stuff. You're probably doing that already, but sometimes one forgets. :)

For taretting specific files with the find, use this:

find . -name "*.php" -exec iconv -f ISO-8859-1 -t UTF-8 {} -o {}.new \; 

Note the " \;" (that's _space_\; ... It's essential. )

As for the rename, I'm not sure, never used that program before, and I don't know the syntax for it. but, what I do is I do much the same as above, and I:

 find . -name "*.php" -exec mv {} {}.php.new \; 

That mv's each .php file to filename.php.new

The for loop that I wrote, it uses the variable "$i" as the filename... so, for each entity in the `ls`, it will perform the command you specify in the "do" portion, and you reference that entity with a $i. (It can be any variable name, of course). I mention this because in the rename syntax you posted, I didn't see it use a $i, although you were using it as your variable.

I hope what I wrote makes sense. :)

 find . -name "*.php" -exec \;