Files that do not agree on their encoding

A month of exports can arrive in three different encodings: one tool writes UTF-8, another writes Windows-1252, a legacy system writes GBK, and a Windows application writes UTF-16. Concatenating them byte-for-byte produces a file where part of the text is unreadable.

Download free trialBuy TextMerger

How to merge files with different encodings

  1. Open TextMerger and click New Task.
  2. Add the files, or Add Folder... to take a whole directory at once.
  3. Set Encoding: to the encoding you want the merged file to use - UTF-8 is a safe default.
  4. If automatic detection gets one file wrong, set Source: for it (for example GBK for a Chinese export).
  5. Click Merge and choose the output file.

TextMerger with the output encoding and per-file source encoding options

Why a plain concatenation is not enough

cat, copy /b or type join bytes and nothing else. If the input files do not share one encoding, the result is a file no single encoding can read: UTF-8 text sitting next to GBK text sitting next to UTF-16 text. Encoding conversion is the part that has to happen while the files are put together.

What TextMerger does

TextMerger merging a GBK file with UTF-8 output: Source Encoding CP936, Output Encoding UTF-8

What it does not do

It converts encodings; it does not repair text that was already decoded wrongly before it reached you. If a file is not valid in any encoding, no converter can invent the missing bytes.

Related tools

Converting one file to another format (not just encoding): DataFileConverter. Opening a very large merged file: BigTextFileViewer. Full reference: merging text files. Related use case: removing duplicate lines while merging.