UTF-8 explained

What UTF-8 does

Unicode assigns each character a number, its code point. A file or network message, however, is just a sequence of bytes, values from 0 to 255. UTF-8 is the rule for turning code points into bytes and back again. It was designed by Ken Thompson and Rob Pike in 1992 and is defined for the internet in RFC 3629.

Today it is the default almost everywhere: HTML5 pages, JSON, most programming languages’ source files, Linux and macOS file systems, and the vast majority of websites. If you are unsure which encoding to use for anything new, the answer is UTF-8.

The byte patterns

UTF-8 is variable-length. Small code points use few bytes and large ones use more. The first bits of each byte say how long the sequence is:

The x bits are the code point itself, split across the bytes. Continuation bytes always start with 10, so a program that lands in the middle of a character can skip forward to the next boundary. That self-synchronising design is one reason UTF-8 is so robust.

A worked example: the degree sign

The degree sign ° is U+00B0, which is 176 in decimal and 000 1011 0000 in binary (11 bits). That is above U+007F, so it needs two bytes:

1. Split the 11 bits into 5 + 6: 00010 and 110000.
2. Put the first group after 110: 11000010 = C2.
3. Put the second group after 10: 10110000 = B0.

So ° is stored as the two bytes C2 B0. Every symbol page on SymbolHub lists these UTF-8 bytes in its code table, and the Unicode converter shows them for any text you paste.

Why UTF-8 is backwards compatible with ASCII

The first 128 code points are the ASCII characters, and UTF-8 stores them as the same single bytes ASCII always used. A plain English text file is therefore byte-for-byte identical in ASCII and UTF-8. Old tools that only understand ASCII still work on it, and no byte in a multi-byte sequence can be mistaken for an ASCII character, because those bytes are always 128 or above.

Mojibake: why é becomes é

Garbled text, known by the Japanese term mojibake, almost always means UTF-8 bytes were read as a single-byte encoding such as Windows-1252 or Latin-1. Each byte is then shown as its own character:

The fix is to declare the encoding correctly, not to search and replace. For a web page, the first line of the <head> should be <meta charset="utf-8">, and the server should send Content-Type: text/html; charset=utf-8. For a CSV opened in Excel, use Data → From Text/CSV and choose 65001: Unicode (UTF-8) as the file origin. If the damage has already been saved, the text has to be decoded as Windows-1252 and re-encoded as UTF-8, which many editors can do with “Reopen with encoding”.

The opposite mistake, reading Windows-1252 text as UTF-8, shows up as the replacement character � (U+FFFD), a question mark in a diamond. That means a byte was not valid UTF-8.

The byte order mark (BOM)

Some programs, notably older versions of Windows Notepad, write the three bytes EF BB BF at the start of a UTF-8 file. This is the byte order mark, the character U+FEFF encoded in UTF-8. UTF-8 has no byte order, so the BOM only acts as a signature saying “this file is UTF-8”.

It is harmless in most editors but can cause trouble elsewhere: a stray invisible character at the top of a PHP page, a broken first column name in a CSV, or a shell script that will not run. Save as “UTF-8” rather than “UTF-8 with BOM” unless a specific program, such as older Excel, needs the signature to detect the encoding.

How UTF-8 compares with UTF-16

UTF-16 stores most characters in two bytes and the rest, including emoji, in four bytes as a surrogate pair. It is used inside Windows, Java and JavaScript. For English and most European text UTF-8 is smaller; for Chinese or Japanese text UTF-16 can be slightly smaller. On the web, UTF-8 won because of its ASCII compatibility and because it has no byte-order ambiguity.

The difference matters when you count characters in code. In JavaScript, "😀".length is 2, because the string holds two UTF-16 units. In UTF-8 the same emoji is 4 bytes. Our Unicode in CSS and JavaScript guide shows how to count what a reader actually sees.

A UTF-8 checklist

• Save source files, HTML, CSS, JSON and CSV as UTF-8 without a BOM.
• Declare <meta charset="utf-8"> within the first 1024 bytes of every HTML page.
• Configure databases for full UTF-8. In MySQL that means utf8mb4; the older utf8 type cannot store four-byte characters such as emoji.
• When text looks garbled, identify which encoding it was written in before trying to fix it.

With that in place, any symbol from our symbol collections will survive being copied, saved, uploaded and emailed.