UTF-16
   HOME

TheInfoList



OR:

UTF-16 (
16-bit 16-bit microcomputers are microcomputers that use 16-bit microprocessors. A 16-bit register can store 216 different values. The range (computer programming), range of integer values that can be stored in 16 bits depends on the Integer (computer ...
Unicode Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, ...
Transformation Format) is a
character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, Data communication, transmi ...
capable of encoding all 1,112,064 valid
code point In character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, D ...
s of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit ''code units''. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding, now known as
UCS-2 The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the international standard International Organization for Standardization, ISO/International Electrotechnical Commission, IEC  ...
(for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed. UTF-16 is used by systems such as the
Microsoft Windows Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporatio ...
API, the
Java programming language Java is a High-level programming language, high-level, class-based programming, class-based, object-oriented programming, object-oriented programming language that is designed to have as few implementation Dependency (computer science), depende ...
and
JavaScript JavaScript (), often abbreviated as JS, is a programming language that is one of the core technologies of the World Wide Web The World Wide Web (WWW), commonly known as the Web, is an information system enabling documents and oth ...
/ECMAScript. It is also sometimes used for
plain text In computing, plain text is a loose term for data (e.g. file contents) that represent only character (computing), characters of readable material but not its graphical representation nor other objects (Floating point numbers, floating-point numb ...
and word-processing data files on Microsoft Windows. It is rarely used for files on
Unix-like A Unix-like (sometimes referred to as UN*X or *nix) operating system is one that behaves in a manner similar to a Unix system, although not necessarily conforming to or being certified to any version of the Single UNIX Specification. A Unix- ...
systems. UTF-16 is often claimed to be more space-efficient than
UTF-8 UTF-8 is a variable-length character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
for East Asian languages, since it uses two bytes for characters that take 3 bytes in UTF-8. Since real text contains many spaces, numbers, punctuation, markup, and control characters, which take only one byte in UTF-8, this is only true for artificially constructed dense blocks of text. In addition
GB 18030 GB 18030 is a Guobiao standards, Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China. GB18030 is the reg ...
(which supports all of Unicode as well) is always shorter than UTF-16. UTF-16 is the only web-encoding incompatible with
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because of ...
and never gained popularity on the web, where it is declared by under 0.002% (little over 1 thousandth of 1 percent) of web pages, and even then UTF-8 is often used, even though UTF-16 is (also) specified (i.e. because of "contradictory character encoding specifications" and/or "incorrect character encoding defined").
UTF-8 UTF-8 is a variable-length character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
, by comparison, accounts for 98% of all web pages. The Web Hypertext Application Technology Working Group (WHATWG) considers UTF-8 "the mandatory encoding for all ext and that for security reasons browser applications should not use UTF-16. It is used by SMS (i.e. the variable-length UTF-16 needed to support all
emoji An emoji ( ; plural emoji or emojis) is a pictogram, logogram, ideogram or smiley embedded in text and used in electronic messages and web pages. The primary function of emoji is to fill in emotional cues otherwise missing from typed conv ...
characters, the SMS standard specifies its predecessor fixed-width
UCS-2 The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the international standard International Organization for Standardization, ISO/International Electrotechnical Commission, IEC  ...
which do not support most of them).


History

In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" ( UCS) that would replace earlier language-specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (216) values, which would require 2 bytes (16 bits) per character. Two groups worked on this in parallel,
ISO/IEC JTC 1/SC 2 ISO/IEC JTC 1/SC 2 Coded character sets is a standardization Standardization or standardisation is the process of implementing and developing technical standards based on the consensus of different parties that include firms, users, interest gro ...
and the
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) non-profit organization incorporated and based in Mountain View, California, Mountain View, California. Its primary purpose is to maintain and publish the Unicode Standard which was d ...
, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2". When it became increasingly clear that 216 characters would not suffice,
IEEE The Institute of Electrical and Electronics Engineers (IEEE) is a 501(c)(3) organization, 501(c)(3) professional association for electronic engineering and electrical engineering (and associated disciplines) with its corporate office in New Yor ...
introduced a larger 31-bit space and an encoding ( UCS-4) that would require 4 bytes per character. This was resisted by the
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) non-profit organization incorporated and based in Mountain View, California, Mountain View, California. Its primary purpose is to maintain and publish the Unicode Standard which was d ...
, both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise and introduced with version 2.0 of the Unicode standard in July 1996. It is fully specified in RFC 2781, published in 2000 by the
IETF The Internet Engineering Task Force (IETF) is a standards organization for the Internet standard, Internet and is responsible for the technical standards that make up the Internet protocol suite (TCP/IP). It has no formal membership roster or ...
. In the UTF-16 encoding, code points less than 216 are encoded with a single 16-bit code unit equal to the numerical value of the code point, as in the older UCS-2. The newer code points greater than or equal to 216 are encoded by a compound value using ''two'' 16-bit code units. These two 16-bit code units are chosen from the UTF-16 surrogate range which had not previously been assigned to characters. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points. A UTF-16 stream, therefore, consists of single 16-bit code points outside the surrogate range for code points in the
Basic Multilingual Plane In the Unicode Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing syste ...
(BMP), and pairs of 16-bit values within the surrogate range for code points above the BMP. UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard." UTF-16 will never be extended to support a larger number of code points or to support the code points that were replaced by surrogates, as this would violate the Unicode Stability Policy with respect to general category or surrogate code points. (Any scheme that remains a
self-synchronizing code In coding theory, especially in telecommunications, a self-synchronizing code is a uniquely decodable code in which the symbol (data), symbol stream formed by a portion of one code word, or by the overlapped portion of any two adjacent code word ...
would require allocating at least one BMP code point to start a sequence. Changing the purpose of a code point is disallowed.)


Description

Each Unicode code point is encoded either as one or two 16-bit ''code units''. How these 16-bit codes are stored as bytes then depends on the
endianness In computing, endianness, also known as byte sex, is the order or sequence of bytes of a word (data type), word of digital data in computer memory. Endianness is primarily expressed as big-endian (BE) or little-endian (LE). A big-endian system s ...
of the text file or communication protocol. A "character" may need from as few as two bytes to fourteen or even more bytes to be recorded. For instance an emoji flag character takes 8 bytes, since it is "constructed from a pair of Unicode scalar values" (and those values are outside the BMP and require 4 bytes each).


U+0000 to U+D7FF and U+E000 to U+FFFF

Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the
Basic Multilingual Plane In the Unicode Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing syste ...
(BMP) are the ''only'' code points that can be represented in UCS-2. As of Unicode 9.0, some modern non-Latin Asian, Middle-Eastern, and African scripts fall outside this range, as do most
emoji An emoji ( ; plural emoji or emojis) is a pictogram, logogram, ideogram or smiley embedded in text and used in electronic messages and web pages. The primary function of emoji is to fill in emotional cues otherwise missing from typed conv ...
characters.


Code points from U+010000 to U+10FFFF

Code points from the other planes (called Supplementary Planes) are encoded as two 16-bit ''code units'' called a ''surrogate pair'', by the following scheme: * 0x10000 is subtracted from the code point ''(U)'', leaving a 20-bit number ''(U')'' in the hex number range 0x00000–0xFFFFF. Note for these purposes, ''U'' is defined to be no greater than 0x10FFFF. * The high ten bits (in the range 0x000–0x3FF) are added to 0xD800 to give the first 16-bit ''code unit'' or ''high surrogate'' ''(W1)'', which will be in the range . * The low ten bits (also in the range 0x000–0x3FF) are added to 0xDC00 to give the second 16-bit ''code unit'' or ''low surrogate'' ''(W2)'', which will be in the range . Illustrated visually, the distribution of ''U between ''W1'' and ''W2'' looks like: U' = yyyyyyyyyyxxxxxxxxxx // U - 0x10000 W1 = 110110yyyyyyyyyy // 0xD800 + yyyyyyyyyy W2 = 110111xxxxxxxxxx // 0xDC00 + xxxxxxxxxx The ''high surrogate'' and ''low surrogate'' are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8. Since the ranges for the ''high surrogates'' (), ''low surrogates'' (), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are disjoint, it is not possible for a surrogate to match a BMP character, or for two adjacent ''code units'' to look like a legal ''surrogate pair''. This simplifies searches a great deal. It also means that UTF-16 is ''self-synchronizing'' on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units (i.e. the type of ''code unit'' can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as
Shift JIS Shift JIS (Shift Japanese Industrial Standards, also SJIS, MIME name Shift_JIS, known as PCK in Solaris contexts) is a character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), c ...
and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string. UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte. Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. ). The Supplementary Planes contain
emoji An emoji ( ; plural emoji or emojis) is a pictogram, logogram, ideogram or smiley embedded in text and used in electronic messages and web pages. The primary function of emoji is to fill in emotional cues otherwise missing from typed conv ...
s, historic scripts, less used symbols, less used Chinese ideographs, etc. Since the encoding of Supplementary Planes contains 20 significant bits (10 of 16 bits in each of the ''high'' and ''low surrogates''), 220 code points can be encoded, divided into 16 planes of 216 code points each. Including the separately-handled Basic Multilingual Plane, there are a total of 17 planes.


U+D800 to U+DFFF

The Unicode standard reserves these code point values for the high and low surrogates, and they will never be assigned a character, so there should be no reason to encode them. The official Unicode standard says that no UTF forms, including UTF-16, can encode these code points. However, Windows allows unpaired surrogates in filenames and other places, which generally means they have to be supported by software in spite of their exclusion from the Unicode standard. UCS-2, UTF-8, and
UTF-32 UTF-32 (32-bit Unicode transformation format, Unicode Transformation Format) is a fixed-length Character encoding, encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must ...
can encode these code points in trivial and obvious ways, and a large amount of software does so, even though the standard states that such arrangements should be treated as encoding errors. It is possible to unambiguously encode an ''unpaired surrogate'' (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this then when translating between encodings.


Examples

To encode U+10437 (𐐷) to UTF-16: * Subtract 0x10000 from the code point, leaving 0x0437. * For the high surrogate, shift right by 10 (divide by 0x400), then add 0xD800, resulting in 0x0001 + 0xD800 = 0xD801. * For the low surrogate, take the low 10 bits (remainder of dividing by 0x400), then add 0xDC00, resulting in 0x0037 + 0xDC00 = 0xDC37. To decode U+10437 (𐐷) from UTF-16: * Take the high surrogate (0xD801) and subtract 0xD800, then multiply by 0x400, resulting in 0x0001 × 0x400 = 0x0400. * Take the low surrogate (0xDC37) and subtract 0xDC00, resulting in 0x37. * Add these two results together (0x0437), and finally add 0x10000 to get the final decoded UTF-32 code point, 0x10437. The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.


Byte-order encoding schemes

UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the
endianness In computing, endianness, also known as byte sex, is the order or sequence of bytes of a word (data type), word of digital data in computer memory. Endianness is primarily expressed as big-endian (BE) or little-endian (LE). A big-endian system s ...
(byte order) of the computer architecture. To assist in recognizing the byte order of code units, UTF-16 allows a
byte order mark The byte order mark (BOM) is a particular usage of the special Unicode character, , whose appearance as a Magic number (programming)#Magic numbers in files, magic number at the start of a text stream can signal several things to a Computer progra ...
(BOM), a code point with the value U+FEFF, to precede the first actual coded value.UTF-8 encoding produces byte values strictly less than 0xFE, so either byte in the BOM sequence also identifies the encoding as UTF-16 (assuming that UTF-32 is not expected). (U+FEFF is the invisible zero-width non-breaking space/ZWNBSP character.)Use of U+FEFF as the character ZWNBSP instead of as a BOM has been deprecated in favor of U+2060 (WORD JOINER); se
Byte Order Mark (BOM) FAQ
at unicode.org. But if an application interprets an initial BOM as a character, the ZWNBSP character is invisible, so the impact is minimal.
If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values. If the BOM is missing, RFC 2781 recommends section 4.3 says that if there is no BOM, "the text SHOULD be interpreted as being big-endian." According to section 1.2, the meaning of the term "SHOULD" is governed by . In that document, section 3 says "... there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course". that big-endian (BE) encoding be assumed. In practice, due to Windows using little-endian (LE) order by default, many applications assume little-endian encoding. It is also reliable to detect endianness by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it is big-endian. The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically ''not'' supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule. For
Internet The Internet (or internet) is the global system of interconnected computer networks that uses the Internet protocol suite (TCP/IP) to communicate between networks and devices. It is a '' network of networks'' that consists of private, p ...
protocols,
IANA The Internet Assigned Numbers Authority (IANA) is a standards organization that oversees global IP address allocation, Autonomous system (Internet), autonomous system number allocation, DNS root zone, root zone management in the Domain Name Sy ...
has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings (the names are case insensitive). The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols. Similar designations, UCS-2BE and UCS-2LE, are used to show versions of UCS-2.


Usage

UTF-16 is used for text in the OS API of all currently supported versions of
Microsoft Windows Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporatio ...
(and including at least all since
Windows CE Windows Embedded Compact, formerly Windows Embedded CE, Windows Powered and Windows CE, is an operating system subfamily developed by Microsoft as part of its Windows Embedded family of products. Unlike Windows Embedded Standard, which is base ...
/
2000 File:2000 Events Collage.png, From left, clockwise: Protests against Bush v. Gore after the 2000 United States presidential election; Heads of state meet for the Millennium Summit; The International Space Station in its infant form as seen from ST ...
/ XP/
2003 File:2003 Events Collage.png, From top left, clockwise: The crew of STS-107 perished when the Space Shuttle Columbia Space Shuttle Columbia disaster, disintegrated during reentry into Atmosphere of Earth, Earth's atmosphere; SARS became an 2002– ...
/
Vista Vista usually refers to a distant view. Vista may also refer to: Software *Windows Vista, the line of Microsoft Windows client operating systems released in 2006 and 2007 *VistA, (Veterans Health Information Systems and Technology Architecture) ...
/ 7) including
Windows 10 Windows 10 is a major release of Microsoft's Windows NT operating system. It is the direct successor to Windows 8.1, which was released nearly two years earlier. It was Software release cycle#Release to manufacturing (RTM), released to ma ...
. In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages. Older
Windows NT Windows NT is a Proprietary software, proprietary Graphical user interface, graphical operating system produced by Microsoft, the first version of which was released on July 27, 1993. It is a processor-independent, multiprocessing and multi-use ...
systems (prior to Windows 2000) only support
UCS-2 The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the international standard International Organization for Standardization, ISO/International Electrotechnical Commission, IEC  ...
. Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings. While there's been some UTF-8 support for even Windows XP, it was improved (in particular the ability to name a file using UTF-8) in Windows 10 insider build 17035 and the May 2019 update. As of May 2019, Microsoft recommends software use UTF-8 instead of other 8-bit encodings. It is unclear if they are recommending usage of UTF-8 over UTF-16. The
IBM i IBM i (the ''i'' standing for ''integrated'') is an operating system developed by IBM for IBM Power Systems. It was originally released in 1988 as OS/400, as the sole operating system of the IBM AS/400 line of systems. It was renamed to i5/OS in 2 ...
operating system designates
CCSID A CCSID (coded character set identifier) is a 16-bit number that represents a particular character encoding, encoding of a specific code page. For example, Unicode is a code page that has several encoding (so called "transformation") forms, like UT ...
(
code page In computing, a code page is a character encoding and as such it is a specific association of a set of printable character (computing), characters and control characters with unique numbers. Typically each number represents the binary value in a s ...
) 13488 for UCS-2 encoding and CCSID 1200 for UTF-16 encoding, though the system treats them both as UTF-16. UTF-16 is used by the Qualcomm BREW operating systems; the
.NET The domain name A domain name is a String (computer science), string that identifies a realm of administrative autonomy, authority or control within the Internet. Domain names are often used to identify services provided through the Interne ...
environments; and the Qt cross-platform graphical
widget toolkit A widget toolkit, widget library, GUI toolkit, or UX library is a library (computing), library or a collection of libraries containing a set of graphical control elements (called ''widgets'') used to construct the graphical user interface (GUI) of ...
.
Symbian OS Symbian is a discontinued mobile operating system (OS) and computing platform designed for smartphones. It was originally developed as a proprietary software OS for personal digital assistants in 1998 by the Symbian Ltd. consortium. Symbian OS ...
used in Nokia S60 handsets and Sony Ericsson UIQ handsets uses UCS-2. iPhone handsets use UTF-16 for
Short Message Service Short Message/Messaging Service, commonly abbreviated as SMS, is a text messaging service component of most telephone, Internet and mobile device systems. It uses standardized communication protocols that let mobile devices exchange short text ...
instead of UCS-2 described in the 3GPP TS 23.038 (
GSM The Global System for Mobile Communications (GSM) is a standard developed by the European Telecommunications Standards Institute (ETSI) to describe the protocols for second-generation (2G) digital cellular networks used by mobile devices such as ...
) and IS-637 (
CDMA Code-division multiple access (CDMA) is a channel access method used by various radio communication technologies. CDMA is an example of channel access method, multiple access, where several transmitters can send information simultaneously over ...
) standards. The Joliet file system, used in
CD-ROM A CD-ROM (, compact disc read-only memory) is a type of read-only memory consisting of a pre-pressed optical compact disc that contains computer data storage, data. Computers can read—but not write or erase—CD-ROMs. Some CDs, called enhanced ...
media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name). The
Python Python may refer to: Snakes * Pythonidae The Pythonidae, commonly known as pythons, are a Family (biology), family of Venomous snake, nonvenomous snakes found in Africa, Asia, and Australia. Among its members are some of the largest snakes i ...
language environment officially only used UCS-2 internally since version 2.0, but the UTF-8 decoder to "Unicode" produces correct UTF-16. Since Python 2.2, "wide" builds of Unicode are supported which use
UTF-32 UTF-32 (32-bit Unicode transformation format, Unicode Transformation Format) is a fixed-length Character encoding, encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must ...
instead; these are primarily used on Linux. Python 3.3 no longer ever uses UTF-16, instead an encoding that gives the most compact representation for the given string is chosen from ASCII/Latin-1, UCS-2, and UTF-32.
Java Java (; id, Jawa, ; jv, ꦗꦮ; su, ) is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea to the north. With a population of 151.6 million people, Java is the world's List ...
originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0.
JavaScript JavaScript (), often abbreviated as JS, is a programming language that is one of the core technologies of the World Wide Web The World Wide Web (WWW), commonly known as the Web, is an information system enabling documents and oth ...
may use UCS-2 or UTF-16. As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective. In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style "\uXXXX" syntax explicitly limits itself to 4 hex digits. The following examples illustrate the syntax for the non-BMP character "𝄞" (U+1D11E, MUSICAL SYMBOL G CLEF): * The most common (used by C++, C#, D, and several other languages) is to use an upper-case 'U' with 8 hex digits such as "\U0001D11E". * In Java 7 regular expressions, ICU, and Perl, the syntax "\x" must be used; similarly, in
ECMAScript ECMAScript (; ES) is a JavaScript standard intended to ensure the interoperability of web pages across different web browser, browsers. It is standardized by Ecma International in the documenECMA-262 ECMAScript is commonly used for client-side ...
2015 (JavaScript), the escape format is "\u". * In many other cases (such as Java outside of regular expressions), the only way to get non-BMP characters is to enter the surrogate halves individually, for example: "\uD834\uDD1E" for U+1D11E. String implementations based on UTF-16 typically define lengths of the string and allow indexing in terms of these 16-bit ''code units'', not in terms of code points. Neither code points nor code units correspond to anything an end user might recognize as a “character”; the things users identify as characters may in general consist of a base code point and a sequence of combining characters (or might be a sequence of code points of some other kind, for example Hangul conjoining jamos) Unicode refers to this construct as a ''grapheme cluster'' and as such, applications dealing with Unicode strings, whatever the encoding, must cope with the fact that this limits their ability to arbitrarily split and combine strings. UCS-2 is also supported by the
PHP PHP is a general-purpose scripting language geared toward web development. It was originally created by Danish-Canadian programmer Rasmus Lerdorf in 1993 and released in 1995. The PHP reference implementation is now produced by The PHP Gr ...
language and
MySQL MySQL () is an Open-source software, open-source relational database management system (RDBMS). Its name is a combination of "My", the name of co-founder Michael Widenius's daughter My, and "SQL", the acronym for Structured Query Language. A rel ...
.
Swift Swift or SWIFT most commonly refers to: * SWIFT, an international organization facilitating transactions between banks ** SWIFT code * Swift (programming language) * Swift (bird), a family of birds It may also refer to: Organizations * SWIFT, a ...
, version 5,
Apple An apple is an edible fruit produced by an apple tree (''Malus domestica''). Apple fruit tree, trees are agriculture, cultivated worldwide and are the most widely grown species in the genus ''Malus''. The tree originated in Central Asia, wh ...
's preferred application language, switched from UTF-16 to UTF-8 as the preferred encoding. While UTF-16 is half as space-efficient than UTF-8 for ASCII, it is more efficient for (some characters of) East Asian languages, the Chinese Unicode encoding standard
GB 18030 GB 18030 is a Guobiao standards, Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China. GB18030 is the reg ...
always produces files the same size or smaller than UTF-16 (or UTF-8) for all languages, not just for Chinese. Additionally, ASCII files can be decoded (are a subset of GB 18030), i.e. ASCII characters are encoded with 1 byte per letter, unlike UTF-16. It takes 2 bytes per letter for e.g.
Devanagari Devanagari ( ; , , Sanskrit pronunciation: ), also called Nagari (),Kathleen Kuiper (2010), The Culture of India, New York: The Rosen Publishing Group, , page 83 is a left-to-right abugida (a type of segmental Writing systems#Segmental syste ...
and Bengali, while in UTF-8 they take 3 bytes.


See also

* Comparison of Unicode encodings *
Plane (Unicode) In the Unicode Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing syste ...
*
UTF-8 UTF-8 is a variable-length character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
*
UTF-32 UTF-32 (32-bit Unicode transformation format, Unicode Transformation Format) is a fixed-length Character encoding, encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must ...


Notes


References


External links


A very short algorithm for determining the surrogate pair for any code point

Unicode Technical Note #12: UTF-16 for Processing


* ttps://www.unicode.org/charts/charindex.html Unicode Character Name Index* : UTF-16, an encoding of ISO 10646
java.lang.String documentation, discussing surrogate handling
{{DEFAULTSORT:Utf-16 Encodings Character encoding Unicode Transformation Formats Computer-related introductions in 1991