Popularity of text encodings

There are many methods of translating text into digital data, such as Baudot code, EBCDIC, and UTF-8, and the relative usage levels of them can provide insight into their usability, and historical trends can show the progress of new methods.

Exact measurements are not possible. Counts of numbers of documents are different than counts weighed by actual use or visibility of those documents. The encoding popularity varies depending on the language used for the documents, or the locale that is the source of the document, or the purpose of the document. Text may be ambiguous as to what encoding it is in, for instance pure ASCII text is valid ASCII or ISO-8859-1 or CP1252 or UTF-8. "Tags" may indicate a document encoding, but when this is incorrect this may be silently corrected by display software (for instance the HTML spec says that the tag for ISO-8859-1 should be treated as CP1252), so counts of tags may not be accurate.

Popularity on the World Wide Web

UTF-8 has been the most common encoding for the World Wide Web since 2008. , UTF-8 accounts for on average 97.8% (previously up to 98.0%) of all web pages (and 99.1% of top 10,000 pages and 984 of the top 1,000 highest ranked web pages, the next most popular encoding, ISO-8859-1, is used by 15 of those sites). Although many pages only use ASCII characters to display content, few websites now declare their encoding to only be ASCII instead of UTF-8.

Virtually all countries and over 97% all of the tracked languages have 95% or more use of UTF-8 encodings on the web. See below for the major alternative encodings:

The second-most popular encoding varies depending on locale, and is typically more efficient for the associated language. One such encoding is the Chinese GB 18030 standard, which is a full Unicode Transformation Format, still 95.0% of websites in China and territories use UTF-8 with it (effectively) the next popular encoding. Big5 is another popular Chinese (for traditional characters) encoding and is next-most popular in Taiwan after UTF-8 at 96.0%, and it's second but used less elsewhere such as in Hong Kong. The single-byte Windows-1251 is twice as efficient for the Cyrillic script and still 93.7% of Russian websites use UTF-8. E.g. Greek and Hebrew encodings are also twice as efficient, and UTF-8 has over 99% use for those languages. Japanese language websites have relatively low UTF-8 use compared to most other countries at 95.0% followed by the legacy Shift JIS and then EUC-JP encoding. South Korea has 95.3% UTF-8 use, with the rest of websites mainly using EUC-KR which is more efficient for Korean text. With the exception of GB 18030 (and UTF-16 and UTF-8), other (legacy) encodings were designed for specific languages, and do not support all Unicode characters. , Breton has the lowest UTF-8 use on the web of any tracked language, with 87% use. Well over a third of the languages tracked have 100.0% use of UTF-8 on the web, such as Vietnamese, Marathi, Telugu, Tamil, Javanese, Pañjābī/Punjabi, Gujarati, Farsi/Persian, Hausa, Pashto, Kannada, Lao, Kurdish languages, Tagalog, Somali, Khmer/Cambodian, isiZulu/Zulu, Turkmen, Tajik (has 4 different scripts), and a lot of the languages with the fewest speakers (often with their own scrips) such as, Armenian, Mongolian ( which has a top-to-bottom script), Maldivian (Thaana), Greenlandic (Kalaallisut) and also sign languages.

Popularity for local text files

Local storage on computers has considerably more use of "legacy" single-byte encodings than on the web. Attempts to update to UTF-8 have been blocked by editors that do not display or write UTF-8 unless the first character in a file is a byte order mark, making it impossible for other software to use UTF-8 without being rewritten to ignore the byte order mark on input and add it on output. UTF-16 files are also fairly common on Windows, but not in other systems.

Popularity internally in software

In the memory of a computer program, usage of UTF-16 is very common, particularly in Windows but and also in JavaScript, Qt, and many other cross-platform software libraries. Compatibility with the Windows API is a major reason for this.

At one time it was believed by many (and is still believed today by some) that having fixed-size code units offers computational advantages, which led many systems, in particular Windows, to use the fixed-size UCS-2 with two bytes per character. This is false: strings are almost never randomly accessed, and sequential access is the same speed. In addition, even UCS-2 was not "fixed size" if combining characters are considered, and when Unicode exceeded 65536 code points it had to be replaced with the non-fixed-sized UTF-16 anyway.

Recently it has become clear that the overhead of translating from/to UTF-8 on input and output, and dealing with potential encoding errors in the input UTF-8, vastly overwhelms any savings UTF-16 could offer. So newer software systems are starting to use UTF-8. International Components for Unicode (ICU) has historically only used UTF-16, and still does only for Java, while it now supports UTF-8 (for C/C++ and other languages indirectly), e.g. used that way by Microsoft; supported as the "Default Charset" including the correct handling of "illegal UTF-8". The default string primitive used in newer programing languages, such as Go, Julia, Rust and Swift 5, assume UTF-8 encoding. PyPy is also using UTF-8 for its strings, and Python is looking into storing all strings with UTF-8. Microsoft now recommends the use of UTF-8 for applications using the Windows API, while continuing to maintain a legacy "Unicode" (meaning UTF-16) interface.

References

{{reflist

Character encoding
Unicode Transformation Formats
Character sets