Sponge, part of the Moreutils package, is a neat perl script which takes data from stdin and writes it to a file. This in itself this is not remarkable, after all you can use the '>' operator to do it. But what is different, is how Sponge waits until the end-of-file character (EOF) before opening and writing to an output file. I.e. it soaks up all the input data before commencing writing.

This is very, very handy if you want to do in-place substitutions with something like sed or in this case, the GNU text encoding conversion utility iconv.

Typically to convert between two encodings, you call iconv like this:

$ iconv -f cp1252 -t utf-8 myfile.txt

This will convert myfile.txt from windows latin 1, to utf-8 and dump the results to stdout.(Don't make the mistake of thinking iso-8859-1 and cp1252 are equivalent. It is safe to convert iso-8859-1 content as if it's cp1252, but the reverse is NOT true).

If you're in any doubt as to the encoding of the source file, you can inspect it with the 'file' command.

So if we have a directory of say c source files we want to convert, we can make use of bash, iconv and sponge to save us the tedium of converting each file manually to a new copy, then replacing the original file with the copy.

$ for FILE in *.c *.h; do iconv -f cp1252 -t utf-8 "$FILE" | sponge "$FILE"

Each file is filtered through iconv, the outputs of which are piped into sponge. Sponge soaks up the standard input until the EOF, then writes it to the original file.