HOME

TheInfoList




UTF-32 (32-
bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algo ...
Unicode Transformation Format Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requireme ...
) is a fixed-length encoding used to encode Unicode
code point In character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, D ...
s that uses exactly 32 bits (four
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
s) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode code points, needing actually only 21 bits). UTF-32 is a fixed-length encoding, in contrast to all other Unicode transformation formats, which are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value. The main advantage of UTF-32 is that the Unicode code points are directly indexed (but letters in general, i.e. "
grapheme In linguistics Linguistics is the scientific study of language A language is a structured system of communication used by humans, including speech (spoken language), gestures (Signed language, sign language) and writing. Most langu ...

grapheme
clusters", or some
emoji An emoji ( ; plural emoji or emojis) is a pictogram A pictogram, also called a pictogramme, pictograph, or simply picto, and in computer usage an icon An icon (from the Greek language, Greek 'image, resemblance') is a religious wor ...

emoji
s can't be directly indexed, nor is calculating the displayed width of a string easier). Finding the ''Nth'' code point in a sequence of code points is a
constant time In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of Algor ...
operation. In contrast, a
variable-length code In coding theory Coding theory is the study of the properties of s and their respective fitness for specific applications. Codes are used for , , , and . Codes are studied by various scientific disciplines—such as , , , , and —for the pur ...
requires
sequential access Sequential access is a term describing a group of elements (such as data in a memory array or a Hard disk drive, disk file or on magnetic tape data storage) being accessed in a predetermined, ordered sequence. It is the opposite of random access, t ...
to find the ''Nth'' code point in a sequence. This makes UTF-32 a simple replacement in code that uses integers that are incremented by one to examine each location in a string, as was commonly done for
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
. The main disadvantage of UTF-32 is that it is space-inefficient, using four
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
s per code point, including 11 bits that are always zero. Characters beyond the BMP are relatively rare in most texts (except for e.g. texts with some popular emojis), and can typically be ignored for sizing estimates. This makes UTF-32 close to twice the size of
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
. It can be up to four times the size of
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
depending on how many of the characters are in the
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
subset.


History

The original
ISO 10646 The International Organization for Standardization (ISO ) is an international standard An international standard is a technical standard A technical standard is an established norm Norm, the Norm or NORM may refer to: In academic discipline ...
standard defines a 32-bit ''encoding form'' called UCS-4, in which each code point in the
Universal Character Set The Universal Coded Character Set (UCS, Unicode) is a standard set of characters Character(s) may refer to: Arts, entertainment, and media Literature * Character (novel), ''Character'' (novel), a 1936 Dutch novel by Ferdinand Bordewijk * Ch ...
(UCS) is represented by a 31-bit value from 0 to 0x7FFFFFFF (the sign bit was unused and zero). In November 2003, Unicode was restricted by RFC 3629 to match the constraints of the
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32.Mapping code points to Unicode encoding forms
§ 1: UTF-32
Although the ISO standard had (as of 1998 in Unicode 2.1) "reserved for private use" 0xE00000 to 0xFFFFFF, and 0x60000000 to 0x7FFFFFFF these areas were removed in later versions. Because the Principles and Procedures document of
ISO/IEC JTC 1/SC 2ISO/IEC JTC 1/SC 2 Coded character sets is a standardization Standardization or standardisation is the process of implementing and developing technical standard A technical standard is an established norm or requirement for a repeatable technic ...
Working Group 2 states that all future assignments of code points will be constrained to the Unicode range, UTF-32 will be able to represent all UCS code points and UTF-32 and UCS-4 are identical.


Analysis

Though a fixed number of bytes per code point seems convenient, it is not as useful as it appears. It makes truncation easier but not significantly so compared to
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
and
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
(both of which can search backwards for the point to truncate by looking at 2–4 code units at most). In practice, because of e.g. emoji ZWJ sequences (and/or emoji modifiers changing possibly
skin color Afghan children with fair skin Human skin color ranges from the darkest brown to the lightest hues. Differences in skin color among individuals is caused by variation in pigment A pigment is a colored material that is completely or nearly i ...
), some
emoji An emoji ( ; plural emoji or emojis) is a pictogram A pictogram, also called a pictogramme, pictograph, or simply picto, and in computer usage an icon An icon (from the Greek language, Greek 'image, resemblance') is a religious wor ...

emoji
s are no longer one code point, such as "👨‍🦲 Man: Bald" and "👩‍🦰 Woman: Red Hair" and thus text can't be relied on being a list of symbols each represented by one code point. It is extremely rare that code wishes to find the ''Nth'' code point without earlier examining the code points 0 to N–1. For instance, XML parsing cannot do anything with a character without first looking at all preceding characters. So an integer index that is incremented by 1 for each character can be replaced with an integer offset, measured in code units and incremented by the number of code units as each character is examined. This removes the perceived speed advantages of UTF-32. UTF-32 does not make calculating the displayed width of a string easier, since even with a "fixed width" font there may be more than one code point per character position (
combining character In digital typography, combining characters are Character (computing), characters that are intended to modify other characters. The most common combining characters in the Latin script are the combining diacritic, diacritical marks (including co ...
s) or more than one character position per code point ("
grapheme In linguistics Linguistics is the scientific study of language A language is a structured system of communication used by humans, including speech (spoken language), gestures (Signed language, sign language) and writing. Most langu ...

grapheme
clusters" for
CJK In internationalization, CJK characters is a collective term for the Chinese, Japanese, and Korean Korean may refer to: People and culture * Koreans, an ethnic group originating in the Korean Peninsula * Korean cuisine * Korean culture * Kor ...
ideographs). Editors that limit themselves to left-to-right languages and
precomposed characterA precomposed character (alternatively composite character or decomposable character) is a Unicode Unicode is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of ...
s can take advantage of fixed-sized code units, but such editors are unlikely to support non-BMP characters and thus can work equally well with
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
.


Use

The main use of UTF-32 is in internal APIs where the data is single code points or glyphs, rather than strings of characters. For instance, in modern text rendering, it is common that the last step is to build a list of structures each containing , attributes, and a single UTF-32 code point identifying the glyph to draw. Often non-Unicode information is stored in the "unused" 11 bits of each word. Use of UTF-32 strings on Windows (where is 16 bits) is almost non-existent. On Unix systems, UTF-32 strings are sometimes, but rarely, used internally by applications, due to the type being defined as 32 bit.
Python PYTHON was a Cold War contingency plan of the Government of the United Kingdom, British Government for the continuity of government in the event of Nuclear warfare, nuclear war. Background Following the report of the Strath Committee in 1955, the ...
versions up to 3.2 can be compiled to use them instead of
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
; from version 3.3 onward all Unicode strings are stored in UTF-32 but with leading zero bytes optimized away "depending on the
ode point An ode (from grc, ᾠδή, ōdḗ) is a type of lyric poetry, lyrical stanza. It is an elaborately structured poem praising or glorifying an event or individual, describing nature intellectually as well as emotionally. A classic ode is structured ...
with the largest Unicode ordinal (1, 2, or 4 bytes)" to make all code points that size.
Seed7 Seed7 is an extensible Extensibility is a software engineering and systems design Systems design is the process of defining the architecture File:Plan d'exécution du second étage de l'hôtel de Brionne (dessin) De Cotte 2503c – Galli ...
and
Lasso A lasso ( or ), also called lariat, riata, or reata (all from Castilian Castilian or Castillian may refer to: * Castile (historical region), Castile, a historic region of Spain ** Castilian people, an ethnic group from Castile ** Spanish ...
programming languages encode all strings with UTF-32, in the belief that direct indexing is important, whereas the
Julia Julia is usually a feminine given name. It is a Latinate feminine form of the name Julio (given name), Julio and Julius. (For further details on etymology, see wikt:Iulius#Latin, Wiktionary entry “Julius”.) The given name ''Julia'' had been ...
programming language moved away from builtin UTF-32 support with its 1.0 release, simplifying the language to having only UTF-8 strings (with all the other encodings considered legacy and moved out of the standard library to package) following the "UTF-8 Everywhere Manifesto".


Variants

Though technically invalid, the surrogate halves are often encoded and allowed. This allows invalid UTF-16 (such as Windows filenames) to be translated to UTF-32, similar to how the
WTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ) ...
variant of UTF-8 works. Sometimes paired surrogates are encoded instead of non-BMP characters, similar to
CESU-8 The Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8) is a variant of UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, ...
. Due to the large number of unused 32-bit values, it is also possible to preserve invalid UTF-8 by using non-Unicode values to encode UTF-8 errors, though there is no standard for this.


See also

*
Comparison of Unicode encodings This article compares Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), text expr ...


References


External links


The Unicode Standard 5.0.0, chapter 3
formally defines UTF-32 in § 3.9, D90 (PDF page 40) and § 3.10, D99-D101 (PDF page 45)

formally defined UTF-32 for Unicode 3.x (March 2001; last updated March 2002)

announcement of UTF-32 being added to the IANA charset registry (April 2002) {{DEFAULTSORT:Utf-32 Character encoding Unicode Transformation Formats