HOME
The Info List - UTF-32


--- Advertisement ---



UTF-32 stands for Unicode
Unicode
Transformation Format in 32 bits. It is a protocol to encode Unicode
Unicode
code points that uses exactly 32 bits per Unicode
Unicode
code point (but a number of leading bits must be zero as there are fewer than 221 Unicode
Unicode
code points). UTF-32 is a fixed-length encoding, in contrast to all other Unicode
Unicode
transformation formats, which are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode
Unicode
code point and is exactly equal to that code point's numerical value. The main advantage of UTF-32 is that the Unicode
Unicode
code points are directly indexed. Finding the Nth code point in a sequence of code points is a constant time operation. In contrast, a variable-length code requires sequential access to find the Nth code point in a sequence. This makes UTF-32 a simple replacement in code that uses integers that are incremented by one to examine each location in a string, as was commonly done for ASCII. The main disadvantage of UTF-32 is that it is space-inefficient, using four bytes per code point. Characters beyond the BMP are relatively rare in most texts, and can typically be ignored for sizing estimates. This makes UTF-32 close to twice the size of UTF-16. It can be up to four times the size of UTF-8
UTF-8
depending on how many of the characters are in the ASCII
ASCII
subset.

Contents

1 History 2 Analysis 3 Use 4 Variants 5 See also 6 References 7 External links

History[edit] The original ISO 10646
ISO 10646
standard defines a 32-bit encoding form called UCS-4, in which each code point in the Universal Character Set (UCS) is represented by a 31-bit value between 0 and 0x7FFFFFFF (the sign bit was unused and zero). In November 2003, Unicode
Unicode
was restricted by RFC 3629 to match the constraints of the UTF-16
UTF-16
encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32.[1][2] Although the ISO standard had (as of 1998 in Unicode 2.1) "reserved for private use" 0xE00000 to 0xFFFFFF, and 0x60000000 to 0x7FFFFFFF[3] these areas were removed in later versions. Because the Principles and Procedures document of ISO/IEC JTC 1/SC 2 Working Group 2 states that all future assignments of code points will be constrained to the Unicode
Unicode
range, UTF-32 will be able to represent all UCS code points and UTF-32 and UCS-4 are identical. Analysis[edit] Though a fixed number of bytes per code point appear convenient, it is not as useful as it appears. It makes truncation easier but not significantly so compared to UTF-8
UTF-8
and UTF-16
UTF-16
(both of which can search backwards for the point to truncate by looking at 2–4 code units at most). It is extremely rare[citation needed] that code wishes to find the Nth code point without earlier examining the code points 0 to N–1, for instance XML parsing cannot do anything with a character without first looking at all preceding characters.[4] So an integer index that is incremented by 1 for each character can be replaced with an integer offset, measured in code units and incremented by the number of code units as each character is examined. This removes the perceived speed advantages[citation needed] of UTF-32. UTF-32 does not make calculating the displayed width of a string easier, since even with a "fixed width" font there may be more than one code point per character position (combining characters) or more than one character position per code point ("grapheme clusters" for CJK ideographs). Editors that limit themselves to left-to-right languages and precomposed characters can take advantage of fixed-sized code units, but such editors are unlikely to support non-BMP characters and thus can work equally well with 16-bit UTF-16
UTF-16
encoding. Use[edit] The main use of UTF-32 is in internal APIs where the data is single code points or glyphs, rather than strings of characters. For instance in modern text rendering it is common that the last step is to build a list of structures each containing coordinates (x,y), attributes, and a single UTF-32 code point identifying the glyph to draw. Often non- Unicode
Unicode
information is stored in the "unused" 11 bits of each word.[citation needed] Use of UTF-32 strings on Windows (where wchar_t is 16 bits) is almost non-existent. On Unix systems, UTF-32 strings are sometimes, but rarely, used internally by applications, due to the type wchar_t being defined as 32 bit. Python versions up to 3.2 can be compiled to use them instead of UTF-16; from version 3.3 onward all Unicode
Unicode
strings are stored in UTF-32 but with leading zero bytes optimized away "depending on the [code point] with the largest Unicode
Unicode
ordinal (1, 2, or 4 bytes)" to make all code points that size.[5] Seed7[citation needed] and Lasso[citation needed] programming languages encode all strings with UTF-32, in the belief that direct indexing is important, whereas Julia had UTF-32 as one of the native encodings for strings (in addition to UTF-8
UTF-8
and UTF-16) in the standard library, but simplified to having only UTF-8
UTF-8
strings (with all the other encodings considered legacy and moved out of the standard library to package[6]) in accordance with the " UTF-8
UTF-8
Everywhere Manifesto".[7] Variants[edit] Though technically invalid, the surrogate halves are often encoded and allowed. This allows invalid UTF-16
UTF-16
(such as Windows filenames) to be translated to UTF-32, similar to how the WTF-8
WTF-8
variant of UTF-8
UTF-8
works. Sometimes paired surrogates are encoded instead of non-BMP characters, similar to CESU-8. Due to the large number of unused 32-bit values, it is also possible to preserve invalid UTF-8
UTF-8
by using non- Unicode
Unicode
values to encode UTF-8
UTF-8
errors, though there is no standard for this. See also[edit]

Comparison of Unicode
Unicode
encodings

References[edit]

^ ISO/IEC 10646:2014 Clause 9.4: "Because surrogate code points are not UCS scalar values, UTF-32 code units in the range 0000 D800-0000 DFFF are ill-formed". Clause 4.57: "[UCS codespace] consisting of the integers from 0 to 10 FFFF (hexadecimal)". Clause 4.58: "[UCS scalar value] any UCS code point except high-surrogate and low-surrogate code points". ^ Mapping code points to Unicode
Unicode
encoding forms, § 1: UTF-32 ^ THE UNIVERSAL CHARACTER SET (UCS) ^ https://www.ibm.com/developerworks/xml/library/x-utf8/ ^ Löwis, Martin. "PEP 393 -- Flexible String Representation". python.org. Python. Retrieved 26 October 2014.  ^ https://github.com/JuliaArchive/LegacyStrings.jl ^ " UTF-8
UTF-8
Everywhere Manifesto". 

External links[edit]

The Unicode
Unicode
Standard 5.0.0, chapter 3 – formally defines UTF-32 in § 3.10, D99-D101 Unicode
Unicode
Standard Annex #19 – formally defined UTF-32 for Unicode
Unicode
3.x (March 2001; last updated March 2002) Registration of new charsets: UTF-32, UTF-32BE, UTF-32LE – announcement of UTF-32 being added to the IANA charset registry (April 2002)

v t e

Unicode

Unicode

Unicode
Unicode
Consortium ISO/IEC 10646 (Universal Character Set) Versions

Code points

Blocks Universal Character Set Character charts Character property Planes Private Use Areas

Characters

Special
Special
purpose

BOM Combining Grapheme
Grapheme
Joiner Left-to-right mark / Right-to-left mark Soft hyphen Word joiner Zero-width joiner Zero-width non-joiner Zero-width space

Lists

Characters CJK Unified Ideographs Combining character Duplicate characters Numerals Scripts Spaces Symbols Halfwidth and fullwidth

Processing

Algorithms

Bi-directional text Collation

ISO 14651

Equivalence Variation sequences International Ideographs Core

Comparison

BOCU-1 CESU-8 Punycode SCSU UTF-1 UTF-7 UTF-8 UTF-9/UTF-18 UTF-16/UCS-2 UTF-32/UCS-4 UTF-EBCDIC

On pairs of code points

Combining character Compatibility characters Duplicate characters Equivalence Homoglyph Precomposed character

list

Z-variant Variation sequences Regional Indicator Symbol Fitzpatrick modifiers

Usage

Domain names (IDN) Email Fonts HTML

entity references numeric references

Input International Ideographs Core

Related standards

Common Locale Data Repository (CLDR) GB 18030 ISO/IEC 8859 ISO 15924

Related topics

Anomalies ConScript Unicode
Unicode
Registry Ideographic Rapporteur Group International Components for Unicode People involved with Unicode Han unification

Scripts and symbols in Unicode

Common and inherited scripts

Combining marks Diacritics Punctuation Space Numbers

Modern scripts

Adlam Arabic

diacritics

Armenian Balinese Bamum Batak Bengali Bopomofo Braille Buhid Burmese Canadian Aboriginal Chakma Cham Cherokee CJK Unified Ideographs (Han) Cyrillic Deseret Devanagari Ge'ez Georgian Greek Gujarati Gurmukhī Hangul Hanja Hanunó'o Hebrew

diacritics

Hiragana Javanese Kanji Kannada Katakana Kayah Li Khmer Lao Latin Lepcha Limbu Lisu (Fraser) Lontara Malayalam Masaram Gondi Mende Kikakui Miao (Pollard) Mongolian Mro N'Ko New Tai Lue Newa Nushu Ol Chiki Oriya Osage Osmanya Pahawh Hmong Pau Cin Hau Rejang Samaritan Saurashtra Shavian Sinhala Sorang Sompeng Sundanese Sylheti Nagari Syriac Tagbanwa Tai Le Tai Tham Tai Viet Tamil Telugu Thaana Thai Tibetan Tifinagh Tirhuta Vai Warang Citi Yi

Ancient and historic scripts

Ahom Anatolian hieroglyphs Ancient North Arabian Avestan Bassa Vah Bhaiksuki Brāhmī Carian Caucasian Albanian Coptic Cuneiform Cypriot Egyptian hieroglyphs Elbasan Glagolitic Gothic Grantha Hatran Imperial Aramaic Inscriptional Pahlavi Inscriptional Parthian Kaithi Kharosthi Khojki Khudawadi Linear A Linear B Lycian Lydian Mahajani Mandaic Manichaean Marchen Meetei Mayek Meroitic Modi Multani Nabataean Ogham Old Hungarian Old Italic Old Permic Old Persian cuneiform Old Turkic Palmyrene 'Phags-pa Phoenician Psalter Pahlavi Runic Śāradā Siddham South Arabian Soyombo Tagalog (Baybayin) Takri Tangut Ugaritic Zanabazar Square

Notational scripts

Duployan SignWriting

Symbols

Cultural, political, and religious symbols Currency Mathematical operators and symbols Phonetic symbols (including IPA) Emoji

v t e

Character encodings

Early telecommunications

ASCII ISO/IEC 646 ISO/IEC 6937 T.61 BCDIC Baudot code Morse code

Telegraph code Wabun code

Special
Special
telegraphy codes

Non-Latin Chinese Cyrillic

Needle telegraph codes

ISO/IEC 8859

-1 -2 -3 -4 -5 -6 -7 -8 -9 -10 -11 -12 -13 -14 -15 -16

Bibliographic use

ANSEL ISO 5426 / 5426-2 / 5427 / 5428 / 6438 / 6861 / 6862 / 10585 / 10586 / 10754 / 11822 MARC-8

National standards

ArmSCII BraSCII CNS 11643 ELOT 927 GOST 10859 GB 18030 HKSCS ISCII JIS X 0201 JIS X 0208 JIS X 0212 JIS X 0213 KOI-7 KPS 9566 KS X 1001 PASCII SI 960 TIS-620 TSCII VISCII YUSCII

EUC

CN JP KR TW

ISO/IEC 2022

CN JP KR CCCII

MacOS code pages ("scripts")

Arabic Celtic CentEuro ChineseSimp / EUC-CN ChineseTrad / Big5 Croatian Cyrillic Devanagari Dingbats Esperanto Farsi (Persian) Gaelic Greek Gujarati Gurmukhi Hebrew Iceland Japanese / ShiftJIS Korean / EUC-KR Latin-1 Roman Romanian Sámi Symbol Thai / TIS-620 Turkish Ukrainian

DOS code pages

100 111 112 113 151 152 161 162 163 164 165 166 210 220 301 437 449 489 620 667 668 707 708 709 710 711 714 715 720 721 737 768 770 771 772 773 774 775 776 777 778 790 850 851 852 853 854 855/872 856 857 858 859 860 861 862 863 864/17248 865 866/808 867 868 869 874/1161/1162 876 877 878 881 882 883 884 885 891 895 896 897 898 899 900 903 904 906 907 909 910 911 926 927 928 929 932 934 936 938 941 942 943 944 946 947 948 949 950/1370 951 966 991 1034 1039 1040 1041 1042 1043 1044 1046 1086 1088 1092 1093 1098 1108 1109 1114 1115 1116 1117 1118 1119 1125/848 1126 1127 1131/849 1139 1167 1168 1300 1351 1361 1362 1363 1372 1373 1374 1375 1380 1381 1385 1386 1391 1392 1393 1394 Kamenický Mazovia CWI-2 KOI8 MIK Iran System

IBM AIX
IBM AIX
code pages

367 371 806 813 819 895 896 912 913 914 915 916 919 920 921/901 922/902 923 952 953 954 955 956 957 958 959 960 961 963 964 965 970 971 1004 1006 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1029 1036 1089 1111 1124 1129/1163 1133 1350 1382 1383

IBM Apple MacIntosh emulations

1275 1280 1281 1282 1283 1284 1285 1286

IBM Adobe emulations

1038 1276 1277

IBM DEC emulations

1020 1021 1023 1090 1100 1101 1102 1103 1104 1105 1106 1107 1287 1288

IBM HP emulations

1050 1051 1052 1053 1054 1055 1056 1057 1058

Windows code pages

CER-GS 874/1162 (TIS-620) 932/943 (Shift JIS) 936/1386 (GBK) 950/1370 (Big5) 949/1363 (EUC-KR) 1169 1174 Extended Latin-8 1200 (UTF-16LE) 1201 (UTF-16BE) 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1261 1270 54936 (GB18030)

EBCDIC
EBCDIC
code pages

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37/1140 38 39 40 251 252 254 256 257 258 259 260 264 273/1141 274 275 276 277/1142 278/1143 279 280/1144 281 282 283 284/1145 285/1146 286 287 288 289 290 293 297/1147 298 300 310 320 321 322 330 351 352 353 355 357 358 359 360 361 363 382 383 384 385 386 387 388 389 390 391 392 393 394 395 410 420/16804 421 423 424/8616/12712 425 435 500/1148 803 829 833 834 835 836 837 838/838 839 870/1110/1153 871/1149 875/4971/9067 880 881 882 883 884 885 886 887 888 889 890 892 893 905 918 924 930/1390 931 933/1364 935/1388 937/1371 939/1399 1001 1002 1003 1005 1007 1024 1025/1154 1026/1155 1027 1028 1030 1031 1032 1033 1037 1047 1068 1069 1070 1071 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1087 1091 1097 1112/1156 1113 1122/1157 1123/1158 1130/1164 1132 1136 1137 1150 1151 1152 1159 1165 1166 1278 1279 1303 1364 1376 1377 JEF KEIS

Platform specific

Acorn Adobe Standard Adobe Latin 1 Apple II ATASCII Atari ST BICS Casio calculators CDC CPC DEC Radix-50 DEC MCS/NRCS DG International ELWRO-Junior FIELDATA GEM GEOS GSM 03.38 HP Roman Extension HP Roman-8 HP Roman-9 HP FOCAL HP RPL LICS LMBCS Mattel Aquarius MSX NEC APC NeXT PCW PETSCII Sharp calculators TI calculators TRS-80 Ventura International Ventura Symbol WISCII XCCS ZX80 ZX81 ZX Spectrum

Unicode / ISO/IEC 10646

UTF-1 UTF-7 UTF-8 UTF-16
UTF-16
(UTF-16LE/UTF-16BE) / UCS-2 UTF-32 (UTF-32LE/UTF-32BE) / UCS-4 UTF-EBCDIC GB 18030 BOCU-1 CESU-8 SCSU

Miscellaneous code pages

ABICOMP APL ARIB STD-B24 Cork HZ INIS INIS-8 ISO-IR-111 ISO-IR-182 ISO-IR-200 ISO-IR-201 ISO-IR-209 Johab LGR LY1 OML OMS OMX OT1 OT2 OT3 OT4 T2A T2B T2C T2D T3 T4 T5 TS1 TS3 U X2 SEASCII TACE16 TRON UTF-5 UTF-6 WTF-8

Related topics

Code page Control character (C0 C1) CCSID Character encodings in HTML Charset detection Han unification Hardware ISO 6429/IEC 6429/ANSI X3.64 Mojibake

.