Skip to content

Instantly share code, notes, and snippets.

@RARain
Forked from dogancelik/ansi-utf8-conversion.md
Created June 14, 2022 01:22
Show Gist options
  • Save RARain/1fbfd96402623ab2b0b9e142cb828d94 to your computer and use it in GitHub Desktop.
Save RARain/1fbfd96402623ab2b0b9e142cb828d94 to your computer and use it in GitHub Desktop.
ANSI / UTF-8 (with or without BOM) conversion #Windows

Using Uni2Me

  • It's free but discontinued.

Using UTFCast

  • Proprietary software
  • Allows conversion from ANSI to UTF-8 with or without BOM

Using Notepad++

Using Python Script Plugin

from glob import glob
from Npp import notepad

globPath = "C:\MyFiles\*.txt"

for file in glob(globPath):
  notepad.open(file)
  notepad.runMenuCommand("Encoding", "Convert to UTF-8-BOM")
  notepad.save()
  notepad.close()

Using Macros

  1. Start Macro recording
  2. Select Encoding > Convert to UTF-8-BOM
  3. Select all text and copy it (it's a bug otherwise it will replace file contents with Clipboard content)
  4. Save file and close it

Using Bash

Add BOM to an already encoded UTF-8 file

echo -ne '\xEF\xBB\xBF' > utf8-no-bom.txt

Batch conversion using find and iconv

# Find all .txt files and convert them to UTF-8 (assuming US characters only / ANSI)
find *.txt -exec 'iconv -f CP1252 -t UTF-8  {} > {}'

# all Windows character sets
iconv -l | grep -i windows

Batch conversion using ls and iconv

for i in `ls *.txt`; do
  iconv -f WINDOWS-1252 -t UTF8 $i -o $i.utf8
  mv $i.utf8 $i
done

Change in …; do with in $@; do to create a usable Bash file. (e.g. convert.sh myfile.txt myfile2.txt)

Using Batch

Add BOM to all text files using nkf

for %a in (*.txt) do nkf32 -W8 -w8 --overwrite "%a"

Download binary for WindowsSource code

Note: Change -w8 with -w80 to remove BOM

Batch conversion using for and iconv

for %a in (*.txt) do iconv -f CP1252 -t UTF-8 "%a" > "%a"

Trivial methods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment