HOME

TheInfoList




Unicode, formally the Unicode Standard, is an information technology
standard Standard may refer to: Symbols * Colours, standards and guidons, kinds of military signs * Heraldic flag, Standard (emblem), a type of a large symbol or emblem used for identification Norms, conventions or requirements * Standard (metrology ...
for the consistent
encoding In communication Communication (from Latin Latin (, or , ) is a classical language A classical language is a language A language is a structured system of communication Communication (from Latin ''communicare'', mean ...
, representation, and handling of
text Text may refer to: Written word * Text (literary theory) In literary theory, a text is any object that can be "read", whether this object is a work of literature, a street sign, an arrangement of buildings on a city block, or styles of clothin ...
expressed in most of the world's
writing system A writing system is a method of visually representing verbal communication Communication (from Latin Latin (, or , ) is a classical language A classical language is a language A language is a structured system of communic ...
s. The standard, which is maintained by the
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) A 501(c)(3) organization is a corporation, trust, unincorporated association, or other type of organization exempt from federal income tax under section 501(c)(3) of Title 26 of the ...
, defines 144,697 characters covering 159 modern and historic
scripts Script may refer to: Writing systems * Script, a distinctive writing system, based on a repertoire of specific elements or symbols, or that repertoire * Script (styles of handwriting) * Script (Unicode), historical and modern scripts as organis ...
, as well as symbols,
emoji An emoji ( ; plural emoji or emojis) is a pictogram A pictogram, also called a pictogramme, pictograph, or simply picto, and in computer usage an icon An icon (from the Greek language, Greek 'image, resemblance') is a religious wor ...

emoji
, and non-visual control and formatting codes. The Unicode character repertoire is synchronized with
ISO/IEC 10646 ISO/IEC JTC 1, entitled "Information technology", is a joint technical committee (JTC) of the International Organization for Standardization The International Organization for Standardization (ISO ) is an international standard An internat ...
, each being code-for-code identical with the other. ''The Unicode Standard'', however, includes more than just the base
code.
code.
Alongside the character encodings, the Consortium's official publication includes a wide variety of details about the scripts and how to display them:
normalization Normalization or normalisation refers to a process that makes something more normal or regular. Most commonly it refers to: * Normalization (sociology) or social normalization, the process through which ideas and behaviors that may fall outside of ...

normalization
rules, decomposition,
collation Collation is the assembly of written information into a standard order. Many systems of collation are based on numerical order or alphabetical order Alphabetical order is a system whereby character string In computer programming Comp ...
, rendering, and
bidirectional text A bidirectional text contains two text direction A writing system is a method of visually representing verbal communication, based on a script and a orthography, set of rules regulating its use. While both writing and spoken language, speech ...
display order for multilingual texts, and so on. The ''Standard'' also includes reference
data file A data file is a computer file A computer file is a computer resource for recording data Data (; ) are individual facts, statistics, or items of information, often numeric. In a more technical sense, data are a set of values of qualitativ ...
s and visual charts to help developers and designers correctly implement the repertoire. Unicode's success at unifying character sets has led to its widespread and predominant use in the
internationalization and localization In computing, internationalization and localization (American English, American) or internationalisation and localisation (British English), often abbreviated i18n and L10n, are means of adapting computer software to different languages, regional ...
of computer
software Software is a collection of instructions Instruction or instructions may refer to: Computing * Instruction, one operation of a processor within a computer architecture instruction set * Computer program, a collection of instructions Music * I ...

software
. The standard has been implemented in many recent technologies, including modern
operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating systems (OS) like macOS, Linux, Android (operating system), Android and Mi ...

operating system
s,
XML Extensible Markup Language (XML) is a markup language #REDIRECT Markup language In computer text processing, a markup language is a system for annotation, annotating a document in a way that is Syntax (logic), syntactically distinguishable fro ...

XML
, and most modern
programming language A programming language is a formal language In logic, mathematics, computer science, and linguistics, a formal language consists of string (computer science), words whose symbol (formal), letters are taken from an alphabet (computer science) ...

programming language
s. Unicode can be implemented by different character encodings. The Unicode standard defines Unicode Transformation Formats (UTF):
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
,
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
, and
UTF-32 UTF-32 (32-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorith ...
, and several other encodings. The most commonly used encodings are UTF-8, UTF-16, and the obsolete
UCS-2 The Universal Coded Character Set (UCS, Unicode) is a standard set of characters Character(s) may refer to: Arts, entertainment, and media Literature * Character (novel), ''Character'' (novel), a 1936 Dutch novel by Ferdinand Bordewijk * Ch ...
(a precursor of UTF-16 without full support for Unicode);
GB18030 GB 18030 is a Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China China (), officially the Peop ...
, while not an official Unicode standard, is standardized in China and implements Unicode fully. UTF-8, the dominant encoding on the
World Wide Web The World Wide Web (WWW), commonly known as the Web, is an information system An information system (IS) is a formal, sociotechnical Sociotechnical systems (STS) in organizational development is an approach to complex organizational ...
(used in over 95% of websites , and up to 100% for some languages) and on most
Unix-like A Unix-like (sometimes referred to as UN*X or *nix) operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating s ...
operating systems, uses one
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
(8 
bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algo ...
s) for the first 128
code point In character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, D ...
s, and up to 4 bytes for other characters. The first 128 Unicode code points represent the
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
characters, which means that any ASCII text is also a UTF-8 text. UCS-2 uses two
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
s (16 bits) for each character but can only encode the first 65,536
code point In character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, D ...
s, the so-called
Basic Multilingual Plane In the Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), text expressed in most o ...
(BMP). With 1,112,064 possible Unicode code points corresponding to characters (see
below Below may refer to: *Earth *Ground (disambiguation) *Soil *Floor *Bottom (disambiguation) *Less than *Temperatures below freezing *Hell or underworld People with the surname *Fred Below (1926–1988), American blues drummer *Fritz von Below (1853 ...
) on 17 planes, and with over 144,000 code points defined as of version 14.0, UCS-2 is only able to represent less than half of all encoded Unicode characters. Therefore, UCS-2 is obsolete, though still used in software. UTF-16 extends UCS-2, by using the same
16-bit 16-bit microcomputers are computers in which 16-bit microprocessors were the norm. A 16-bit register can store 216 different values. The range (computer programming), range of integer values that can be stored in 16 bits depends on the Integer ( ...
encoding as UCS-2 for the Basic Multilingual Plane, and a 4-byte encoding for the other planes. As long as it contains no code points in the reserved range U+D800–U+DFFF, a UCS-2 text is valid UTF-16 text. UTF-32 (also referred to as UCS-4) uses four bytes to encode any given code point, but not necessarily any given (loosely speaking, a
grapheme In linguistics Linguistics is the scientific study of language A language is a structured system of communication used by humans, including speech (spoken language), gestures (Signed language, sign language) and writing. Most langu ...

grapheme
), since a user-perceived character may be represented by a (a sequence of multiple code points). Like UCS-2, the number of bytes per code point is fixed, facilitating code point indexing; but unlike UCS-2, UTF-32 is able to encode all Unicode code points. However, because each code point uses four bytes, UTF-32 takes significantly more space than other encodings, and is not widely used. Although UTF-32 has a fixed size for each code point, it is also variable-length with respect to user-perceived characters. Examples include: the
Devanagari Devanagari ( ; , , Sanskrit pronunciation: ), also called Nagari (),Kathleen Kuiper (2010), The Culture of India, New York: The Rosen Publishing Group, , page 83 is a left-to-right abugida, based on the ancient Brahmi script, ''Brāhmī'' sc ...

Devanagari
''kshi'', which is encoded by 4 code points, and national flag emojis, which are composed of two code points. All combining character sequences are graphemes, but there are other sequences of code points that are as well, for example \r\n.


Origin and development

Unicode has the explicit aim of transcending the limitations of traditional character encodings, such as those defined by the
ISO/IEC 8859 ISO/IEC 8859 is a joint ISO The International Organization for Standardization (ISO ) is an international standard An international standard is a technical standard A technical standard is an established norm (social), norm or requirement for ...
standard, which find wide usage in various countries of the world but remain largely incompatible with each other. Many traditional character encodings share a common problem in that they allow bilingual computer processing (usually using
Latin character Latin script, also known as Roman script, is an alphabetic writing system based on the letters of the classical Latin alphabet, derived from a form of the Cumae alphabet, Cumaean Greek version of the Greek alphabet used by the Etruscan civilizat ...
s and the local script), but not multilingual computer processing (computer processing of arbitrary scripts mixed with each other). Unicode, in intent, encodes the underlying characters—
grapheme In linguistics Linguistics is the scientific study of language A language is a structured system of communication used by humans, including speech (spoken language), gestures (Signed language, sign language) and writing. Most langu ...

grapheme
s and grapheme-like units—rather than the variant
glyph The term glyph is used in typography Typography is the art and technique of arranging type to make written language A written language is the representation of a spoken or gestural language A language is a structured system o ...
s (renderings) for such characters. In the case of
Chinese characters Chinese characters, also called ''hanzi'' (), are logogram In a written language A written language is the representation of a spoken or gestural language A language is a structured system of communication used by humans, ...
, this sometimes leads to controversies over distinguishing the underlying character from its variant glyphs (see
Han unification Han unification is an effort by the authors of Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Cha ...
). In text processing, Unicode takes the role of providing a unique —a
number A number is a mathematical object A mathematical object is an abstract concept arising in mathematics. In the usual language of mathematics, an ''object'' is anything that has been (or could be) formally defined, and with which one may do deduct ...

number
, not a glyph—for each character. In other words, Unicode represents a character in an abstract way and leaves the visual rendering (size, shape,
font In metal A metal (from Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ), officially the Hellenic Republic, is a country located in Southeast Europe. Its population is approxim ...

font
, or style) to other software, such as a
web browser A web browser (commonly referred to as a browser) is application software for accessing the World Wide Web. When a User (computing), user requests a web page from a particular website, the web browser retrieves the necessary content from a web ...

web browser
or
word processor A word processor (WP) is a device or computer program that provides for input, editing, formatting, and output of text, often with some additional features. Early word processors were stand-alone devices dedicated to the function, but current ...
. This simple aim becomes complicated, however, because of concessions made by Unicode's designers in the hope of encouraging a more rapid adoption of Unicode. The first 256 code points were made identical to the content of
ISO/IEC 8859-1 ISO/IEC 8859-1:1998, ''Information technology — 8-bit In computer architecture In computer engineering, computer architecture is a set of rules and methods that describe the functionality, organization, and implementation of compu ...
so as to make it trivial to convert existing western text. Many essentially identical characters were encoded multiple times at different code points to preserve distinctions used by legacy encodings and therefore, allow conversion from those encodings to Unicode (and back) without losing any information. For example, the "
fullwidth forms In CJK characters, CJK (Chinese, Japanese and Korean) computing, graphic characters are traditionally classed into fullwidth (in Taiwan and Hong Kong: wikt:全形, 全形; in CJK: wikt:全角, 全角) and halfwidth (in Taiwan and Hong Kong: wikt: ...
" section of code points encompasses a full duplicate of the Latin alphabet because Chinese, Japanese, and Korean (
CJK In internationalization In economics, internationalization or internationalisation is the process of increasing involvement of enterprises in international markets, although there is no agreed definition of internationalization. Internationaliz ...
) fonts contain two versions of these letters, "fullwidth" matching the width of the CJK characters, and normal width. For other examples, see
duplicate characters in Unicode Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), text expressed in most of the wor ...
. Unicode Bulldog Award recipients include many names influential in the development of Unicode and include
Tatsuo Kobayashi is a Japanese web architect who specializes in international standardization. Born and raised in Tokyo, he studied history and philosophy of science Philosophy of science is a branch of philosophy concerned with the foundations, methodol ...
, Thomas Milo,
Roozbeh Pournader
Roozbeh Pournader
,
Ken Lunde Ken Roger Lunde (, born 12 August 1965 in Madison, Wisconsin Madison is the capital Capital most commonly refers to: * Capital letter Letter case (or just case) is the distinction between the letters that are in larger uppercase or ...
, and
Michael Everson Michael Everson (born January 9, 1963) is an American and Irish linguist Linguistics is the scientific study of language A language is a structured system of communication Communication (from Latin Latin (, or , ) is a cla ...
.


History

Based on experiences with the Xerox Character Code Standard (XCCS) since 1980, the origins of Unicode date to 1987, when Joe Becker from
Xerox Xerox Holdings Corporation (; also known simply as Xerox) is an American corporation that sells print and digital document An electronic document is any electronic media content (other than computer program A computer program is a collectio ...
with Lee Collins and Mark Davis from
Apple An apple is an edible fruit In botany Botany, also called , plant biology or phytology, is the science of plant life and a branch of biology. A botanist, plant scientist or phytologist is a scientist who specialises in this fie ...
started investigating the practicalities of creating a universal character set. With additional input from Peter Fenwick and Dave Opstad, Joe Becker published a draft proposal for an "international/multilingual text character encoding system in August 1988, tentatively called Unicode". He explained that " e name 'Unicode' is intended to suggest a unique, unified, universal encoding". In this document, entitled ''Unicode 88'', Becker outlined a
16-bit 16-bit microcomputers are computers in which 16-bit microprocessors were the norm. A 16-bit register can store 216 different values. The range (computer programming), range of integer values that can be stored in 16 bits depends on the Integer ( ...
character model:
Unicode is intended to address the need for a workable, reliable world text encoding. Unicode could be roughly described as "wide-body ASCII" that has been stretched to 16 bits to encompass the characters of all the world's living languages. In a properly engineered design, 16 bits per character are more than sufficient for this purpose.
His original 16-bit design was based on the assumption that only those scripts and characters in modern use would need to be encoded:
Unicode gives higher priority to ensuring utility for the future than to preserving past antiquities. Unicode aims in the first instance at the characters published in modern text (e.g. in the union of all newspapers and magazines printed in the world in 1988), whose number is undoubtedly far below 214 = 16,384. Beyond those modern-use characters, all others may be defined to be obsolete or rare; these are better candidates for private-use registration than for congesting the public list of generally useful Unicodes.
In early 1989, the Unicode working group expanded to include Ken Whistler and Mike Kernaghan of Metaphor, Karen Smith-Yoshimura and Joan Aliprand of RLG, and Glenn Wright of
Sun Microsystems Sun Microsystems, Inc. (Sun for short) was an American technology company that sold computer A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations automatically. M ...
, and in 1990, Michel Suignard and Asmus Freytag from
Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporation which produces Software, computer software, consumer electronics, personal computers, and related services. Its best-know ...

Microsoft
and Rick McGowan of
NeXT NeXT, Inc. (later NeXT Computer, Inc. and NeXT Software, Inc.) was an American technology company that specialized in computer workstations A workstation is a special computer designed for technical or scientific Science () is a syst ...
joined the group. By the end of 1990, most of the work on mapping existing character encoding standards had been completed, and a final review draft of Unicode was ready. The
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) A 501(c)(3) organization is a corporation, trust, unincorporated association, or other type of organization exempt from federal income tax under section 501(c)(3) of Title 26 of the ...
was incorporated in California on 3 January 1991, and in October 1991, the first volume of the Unicode standard was published. The second volume, covering Han ideographs, was published in June 1992. In 1996, a surrogate character mechanism was implemented in Unicode 2.0, so that Unicode was no longer restricted to 16 bits. This increased the Unicode codespace to over a million code points, which allowed for the encoding of many historic scripts (e.g.,
Egyptian hieroglyphs Egyptian hieroglyphs () were the formal writing system A writing system is a method of visually representing verbal communication Communication (from Latin ''communicare'', meaning "to share" or "to be in relation with") is "an apparent ...
) and thousands of rarely used or obsolete characters that had not been anticipated as needing encoding. Among the characters not originally intended for Unicode are rarely used Kanji or Chinese characters, many of which are part of personal and place names, making them rarely used, but much more essential than envisioned in the original architecture of Unicode. The Microsoft TrueType specification version 1.0 from 1992 used the name 'Apple Unicode' instead of 'Unicode' for the Platform ID in the naming table.


Unicode Consortium

The Unicode Consortium is a nonprofit organization that coordinates Unicode's development. Full members include most of the main computer software and hardware companies with any interest in text-processing standards, including
Adobe Adobe (; ) is a building material Building material is material used for construction. Many naturally occurring substances, such as clay, rocks, sand, and wood, even twigs and leaves, have been used to construct buildings. Apart from ...
,
Apple An apple is an edible fruit In botany Botany, also called , plant biology or phytology, is the science of plant life and a branch of biology. A botanist, plant scientist or phytologist is a scientist who specialises in this fie ...
,
Facebook Facebook is an American online social media and social networking service owned by Meta Platforms. Founded in 2004 by Mark Zuckerberg with fellow Harvard College students and roommates Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, an ...
,
Google Google LLC is an American multinational Multinational may refer to: * Multinational corporation, a corporate organization operating in multiple countries * Multinational force, a military body from multiple countries * Multinational stat ...

Google
,
IBM International Business Machines Corporation (IBM) is an American multinational technology company headquartered in Armonk, New York, with operations in over 170 countries. The company began in 1911, founded in Endicott, New York, as the C ...

IBM
,
Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporation which produces Software, computer software, consumer electronics, personal computers, and related services. Its best-know ...

Microsoft
,
Netflix Netflix, Inc. is an American subscription The subscription business model is a business model in which a customer In sales Sales are activities related to selling or the number of goods sold in a given targeted time period. Th ...

Netflix
, and
SAP SE SAP SE () is a German multinational Multinational may refer to: * Multinational corporation, a corporate organization operating in multiple countries * Multinational force, a military body from multiple countries * Multinational state, a sove ...
. Over the years several countries or government agencies have been members of the Unicode Consortium. Presently only the
Ministry of Endowments and Religious Affairs (Oman) The Ministry of Awqaf and Religious Affairs (MARA) is the governmental body in the Sultanate of Oman responsible for overseeing all matters related to awqaf and religious affairs. The current Minister of Awqaf and Religious Affairs is Abdullah ...
is a full member with voting rights. The Consortium has the ambitious goal of eventually replacing existing character encoding schemes with Unicode and its standard Unicode Transformation Format (UTF) schemes, as many of the existing schemes are limited in size and scope and are incompatible with
multilingual Multilingualism is the use of more than one language A language is a structured system of communication Communication (from Latin Latin (, or , ) is a classical language belonging to the Italic languages, Italic branch of t ...
environments.


Scripts covered

Unicode covers almost all scripts (
writing system A writing system is a method of visually representing verbal communication Communication (from Latin Latin (, or , ) is a classical language A classical language is a language A language is a structured system of communic ...
s) in current use today. As of 2021 a total of 159
scripts Script may refer to: Writing systems * Script, a distinctive writing system, based on a repertoire of specific elements or symbols, or that repertoire * Script (styles of handwriting) * Script (Unicode), historical and modern scripts as organis ...
are included in the latest version of Unicode (covering
alphabet An alphabet is a standardized set of basic written symbols A symbol is a mark, sign, or word In linguistics, a word of a spoken language can be defined as the smallest sequence of phonemes that can be uttered in isolation with semanti ...

alphabet
s,
abugida An abugida (, from Ge'ez language, Ge'ez: አቡጊዳ), sometimes known as alphasyllabary, neosyllabary or pseudo-alphabet, is a segmental Writing systems#Segmental writing system, writing system in which consonant-vowel sequences are writt ...
s and
syllabaries In the linguistic Linguistics is the scientific study of language A language is a structured system of communication Communication (from Latin Latin (, or , ) is a classical language belonging to the Italic languages, Itali ...
), although there are still scripts that are not yet encoded, particularly those mainly used in historical, liturgical, and academic contexts. Further additions of characters to the already encoded scripts, as well as symbols, in particular for mathematics and
music Music is the of arranging s in time through the of melody, harmony, rhythm, and timbre. It is one of the aspects of all human societies. General include common elements such as (which governs and ), (and its associated concepts , , and ...
(in the form of notes and rhythmic symbols), also occur. The Unicode Roadmap Committee (
Michael Everson Michael Everson (born January 9, 1963) is an American and Irish linguist Linguistics is the scientific study of language A language is a structured system of communication Communication (from Latin Latin (, or , ) is a cla ...
, Rick McGowan, Ken Whistler, V.S. Umamaheswaran) maintain the list of scripts that are candidates or potential candidates for encoding and their tentative code block assignments on the Unicode Roadmap page of the
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) A 501(c)(3) organization is a corporation, trust, unincorporated association, or other type of organization exempt from federal income tax under section 501(c)(3) of Title 26 of the ...
website. For some scripts on the Roadmap, such as
JurchenJurchen may refer to: * Jurchen people, Tungusic people who inhabited the region of Manchuria until the 17th century ** Haixi Jurchens, a grouping of the Jurchens as identified by the Chinese of the Ming Dynasty ** Jianzhou Jurchens, a grouping of t ...
and
Khitan small script The Khitan small script () was one of two writing systems used for the now-extinct Khitan language Khitan or Kitan ( in large script or in small Small may refer to: Science and technology * SMALL Small may refer to: Science and technology ...
, encoding proposals have been made and they are working their way through the approval process. For other scripts, such as
Mayan Mayan most commonly refers to: * Maya peoples, various indigenous peoples of Mesoamerica and northern Central America * Maya civilization, pre-Columbian culture of Mesoamerica and northern Central America * Mayan languages, language family spoken i ...
(besides numbers) and
Rongorongo Rongorongo (Rapa Nui language, Rapa Nui: ) is a system of glyphs discovered in the 19th century on Easter Island that appears to be writing system, writing or proto-writing. Numerous attempts at Decipherment of rongorongo, decipherment have be ...
, no proposal has yet been made, and they await agreement on character repertoire and other details from the user communities involved. Some modern invented scripts which have not yet been included in Unicode (e.g.,
Tengwar The Tengwar script is an artificial script A constructed script is a new writing system A writing system is a method of visually representing verbal communication Communication (from Latin ''communicare'', meaning "to share") is the ...

Tengwar
) or which do not qualify for inclusion in Unicode due to lack of real-world use (e.g.,
Klingon The Klingons ( ; Klingon language, Klingon: ''tlhIngan'' ) are a fictional species in the science fiction franchise ''Star Trek''. Developed by screenwriter Gene L. Coon in 1967 for the Star Trek: The Original Series, original ''Star Trek'' ('' ...
) are listed in the
ConScript Unicode Registry The ConScript Unicode Registry was a volunteer project to coordinate the assignment of code points in the Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character en ...
, along with unofficial but widely used
Private Use Areas In Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), text expressed in most of t ...
code assignments. There is also a
Medieval Unicode Font Initiative In digital typography Desktop publishing (DTP) is the creation of documents using page layout software on a personal ("desktop") personal computer, computer. It was first used almost exclusively for print publications, but now it also assists i ...
focused on special Latin medieval characters. Part of these proposals have been already included into Unicode.


Script Encoding Initiative

The Script Encoding Initiative, a project run by Deborah Anderson at the
University of California, Berkeley The University of California, Berkeley (UC Berkeley, Berkeley, Cal, or California) is a public In public relations Public relations (PR) is the practice of managing and disseminating information from an individual or an organization ...

University of California, Berkeley
was founded in 2002 with the goal of funding proposals for scripts not yet encoded in the standard. The project has become a major source of proposed additions to the standard in recent years.


Versions

The Unicode Consortium and the
International Organization for Standardization The International Organization for Standardization (ISO ) is an international standard An international standard is a technical standard A technical standard is an established norm Norm, the Norm or NORM may refer to: In academic discipline ...
(ISO) have together developed a shared
repertoire A repertoire () is a list or set of dramas, opera Opera is a form of theatre Theatre or theater is a collaborative form of performing art that uses live performers, usually actor, actors or actresses, to present the experience of a ...
following the initial publication of ''The Unicode Standard'' in 1991; Unicode and the ISO's
Universal Coded Character Set The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the International Organization for Standardization, International Standard ISO/International Electrotechnical Commission, IEC  ...
(UCS) use identical character names and code points. However, the Unicode versions do differ from their ISO equivalents in two significant ways. While the UCS is a simple character map, Unicode specifies the rules, algorithms, and properties necessary to achieve interoperability between different platforms and languages. Thus, ''The Unicode Standard'' includes more information, covering—in depth—topics such as bitwise encoding,
collation Collation is the assembly of written information into a standard order. Many systems of collation are based on numerical order or alphabetical order Alphabetical order is a system whereby character string In computer programming Comp ...
and rendering. It also provides a comprehensive catalog of character properties, including those needed for supporting
bidirectional text A bidirectional text contains two text direction A writing system is a method of visually representing verbal communication, based on a script and a orthography, set of rules regulating its use. While both writing and spoken language, speech ...
, as well as visual charts and reference data sets to aid implementers. Previously, ''The Unicode Standard'' was sold as a print volume containing the complete core specification, standard annexes, and code charts. However, Unicode 5.0, published in 2006, was the last version printed this way. Starting with version 5.2, only the core specification, published as print-on-demand paperback, may be purchased. The full text, on the other hand, is published as a free PDF on the Unicode website. A practical reason for this publication method highlights the second significant difference between the UCS and Unicode—the frequency with which updated versions are released and new characters added. ''The Unicode Standard'' has regularly released annual expanded versions, occasionally with more than one version released in a calendar year and with rare cases where the scheduled release had to be postponed. For instance, in April 2020, only a month after version 13.0 was published, the Unicode Consortium announced they had changed the intended release date for version 14.0, pushing it back six months from March 2021 to September 2021 due to the
COVID-19 pandemic The COVID-19 pandemic is an ongoing global pandemic A pandemic (from , , "all" and , , "local people" the 'crowd') is an of an that has spread across a large region, for instance multiple or worldwide, affecting a substantial numbe ...

COVID-19 pandemic
. Thus far, the following major and minor versions of the Unicode standard have been published. Update versions, which do not include any changes to character repertoire, are signified by the third number (e.g., "version 4.0.1") and are omitted in the table below.


Architecture and terminology


Codespace and Code Points

The Unicode Standard defines a ''codespace,'' a set of numerical values ranging from 0 through 10FFFF 16, called ''
code point In character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be Data storage, stored, D ...
s'' and denoted as through ("U+" plus the code point value in
hexadecimal In mathematics and computing, the hexadecimal (also base 16 or hex) numeral system is a Numeral system#Positional systems in detail, positional numeral system that represents numbers using a radix (base) of 16. Unlike the decimal system repres ...
, prepended with
leading zero In typography, leading ( ) is the space between adjacent lines of type; the exact definition varies. In hand typesetting, leading is the thin strips of lead (or aluminium) that were inserted between lines of type in the composing stick to inc ...
s to a minimum of four digits, ''e. g.'', for the division sign but (''not'' ) for the
Egyptian hieroglyph Egyptian hieroglyphs () were the formal writing system A writing system is a method of visually representing verbal communication Communication (from Latin ''communicare'', meaning "to share" or "to be in relation with") is "an apparent ...
.). Of these 216 + 220 defined code points, the code points from through , which are used to encode surrogate pairs in
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
, are reserved by the Unicode Standard and may not be used to encode valid characters, resulting in a net total of 216 − 211 + 220 = 1,112,064 assignable code points.


Code planes and blocks

The Unicode codespace is divided into seventeen ''planes'', numbered 0 to 16: All code points in the BMP are accessed as a single code unit in
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
encoding and can be encoded in one, two or three bytes in
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
. Code points in Planes 1 through 16 (''supplementary planes'') are accessed as surrogate pairs in UTF-16 and encoded in four bytes in UTF-8. Within each plane, characters are allocated within named '' blocks'' of related characters. Although blocks are an arbitrary size, they are always a multiple of 16 code points and often a multiple of 128 code points. Characters required for a given script may be spread out over several different blocks.


General Category property

Each code point has a single
General CategoryThe Unicode Standard assigns character properties to each code point. These properties can be used to handle "characters" (code points) in processes, like in line-breaking, script direction right-to-left or applying controls. Slightly inconsequently, ...
property. The major categories are denoted: Letter, Mark, Number, Punctuation, Symbol, Separator and Other. Within these categories, there are subdivisions. In most cases other properties must be used to sufficiently specify the characteristics of a code point. The possible General Categories are: Code points in the range U+D800–U+DBFF (1,024 code points) are known as high-surrogate code points, and code points in the range U+DC00–U+DFFF (1,024 code points) are known as low-surrogate code points. A high-surrogate code point followed by a low-surrogate code point form a surrogate pair in
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
to represent code points greater than U+FFFF. These code points otherwise cannot be used (this rule is ignored often in practice especially when not using UTF-16). A small set of code points are guaranteed never to be used for encoding characters, although applications may make use of these code points internally if they wish. There are sixty-six of these noncharacters: U+FDD0–U+FDEF and any code point ending in the value FFFE or FFFF (i.e., U+FFFE, U+FFFF, U+1FFFE, U+1FFFF, ... U+10FFFE, U+10FFFF). The set of noncharacters is stable, and no new noncharacters will ever be defined. Like surrogates, the rule that these cannot be used is often ignored, although the operation of the byte order mark assumes that U+FFFE will never be the first code point in a text. Excluding surrogates and noncharacters leaves 1,111,998 code points available for use. Private-use code points are considered to be assigned characters, but they have no interpretation specified by the Unicode standard so any interchange of such characters requires an agreement between sender and receiver on their interpretation. There are three private-use areas in the Unicode codespace: * Private Use Area: U+E000–U+F8FF (6,400 characters), * Supplementary Private Use Area-A: U+F0000–U+FFFFD (65,534 characters), * Supplementary Private Use Area-B: U+100000–U+10FFFD (65,534 characters). Graphic characters are characters defined by Unicode to have particular semantics, and either have a visible
glyph The term glyph is used in typography Typography is the art and technique of arranging type to make written language A written language is the representation of a spoken or gestural language A language is a structured system o ...
shape or represent a visible space. As of Unicode 14.0 there are 144,532 graphic characters. Format characters are characters that do not have a visible appearance, but may have an effect on the appearance or behavior of neighboring characters. For example, and may be used to change the default shaping behavior of adjacent characters (e.g., to inhibit ligatures or request ligature formation). There are 165 format characters in Unicode 14.0. Sixty-five code points (U+0000–U+001F and U+007F–U+009F) are reserved as control codes, and correspond to the
C0 and C1 control codes The C0 and C1 control code or control character In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm ...
defined in
ISO/IEC 6429 ISO/IEC JTC 1, entitled "Information technology", is a joint technical committee (JTC) of the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Its purpose is to develop, maintain and pr ...
. U+0009 (Tab), U+000A (Line Feed), and U+000D (Carriage Return) are widely used in Unicode-encoded texts. In practice the C1 code points are often improperly-translated (
mojibake Mojibake (; ) is the garbled text that is the result of text being decoded using an unintended character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes ...

mojibake
) as the legacy
Windows-1252 Windows-1252 or CP-1252 (code page In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algorith ...

Windows-1252
characters used by some English and Western European texts. Graphic characters, format characters, control code characters, and private use characters are known collectively as ''assigned characters''. Reserved code points are those code points which are available for use, but are not yet assigned. As of Unicode 14.0 there are 829,768 reserved code points.


Abstract characters

The set of graphic and format characters defined by Unicode does not correspond directly to the repertoire of ''abstract characters'' that is representable under Unicode. Unicode encodes characters by associating an abstract character with a particular code point. However, not all abstract characters are encoded as a single Unicode character, and some abstract characters may be represented in Unicode by a sequence of two or more characters. For example, a Latin small letter "i" with an
ogonek The (; Polish: , "little tail", diminutive of ) is a diacritic hook placed under the lower right corner of a vowel in the Latin alphabet used in several European languages, and directly under a vowel in several Native American languages. It i ...

ogonek
, a , and an
acute accent The acute accent, , is a diacritic A diacritic (also diacritical mark, diacritical point, diacritical sign, or accent) is a glyph The term glyph is used in typography File:metal movable type.jpg, 225px, Movable type being assembl ...

acute accent
, which is required in
Lithuanian Lithuanian may refer to: * Lithuanians * Lithuanian language * The country of Lithuania * Grand Duchy of Lithuania * Culture of Lithuania * Lithuanian cuisine * Lithuanian Jews as often called "Lithuanians" (''Lita'im'' or ''Litvaks'') by other Jew ...
, is represented by the character sequence U+012F, U+0307, U+0301. Unicode maintains a list of uniquely named character sequences for abstract characters that are not directly encoded in Unicode. All graphic, format, and private use characters have a unique and immutable name by which they may be identified. This immutability has been guaranteed since Unicode version 2.0 by the Name Stability policy. In cases where the name is seriously defective and misleading, or has a serious typographical error, a formal alias may be defined, and applications are encouraged to use the formal alias in place of the official character name. For example, has the formal alias , and has the formal alias .


Ready-made versus composite characters

Unicode includes a mechanism for modifying characters that greatly extends the supported glyph repertoire. This covers the use of
combining diacritical mark In digital typography, combining characters are Character (computing), characters that are intended to modify other characters. The most common combining characters in the Latin script are the combining diacritic, diacritical marks (including c ...
s that may be added after the base character by the user. Multiple combining diacritics may be simultaneously applied to the same character. Unicode also contains precomposed versions of most letter/diacritic combinations in normal use. These make conversion to and from legacy encodings simpler, and allow applications to use Unicode as an internal text format without having to implement combining characters. For example, ''é'' can be represented in Unicode as U+0065 () followed by U+0301 (), but it can also be represented as the precomposed character U+00E9 (). Thus, in many cases, users have multiple ways of encoding the same character. To deal with this, Unicode provides the mechanism of
canonical equivalence Unicode equivalence is the specification by the Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Charact ...
. An example of this arises with
Hangul The Korean alphabet, known as Hangul, . Hangul may also be written as following South Korea's standard Romanization. in South Korea South Korea, officially the Republic of Korea (ROK), is a country in East Asia, constituting the ...

Hangul
, the Korean alphabet. Unicode provides a mechanism for composing Hangul syllables with their individual subcomponents, known as
Hangul Jamo The Korean alphabet, known as Hangul, . Hangul may also be written as following South Korea's standard Romanization. in South Korea South Korea, officially the Republic of Korea (ROK), is a country in East Asia, constituting the ...
. However, it also provides 11,172 combinations of precomposed syllables made from the most common jamo. The
CJK characters In internationalization In economics, internationalization or internationalisation is the process of increasing involvement of enterprises in international markets, although there is no agreed definition of internationalization. Internationaliz ...
currently have codes only for their precomposed form. Still, most of those characters comprise simpler elements (called radicals), so in principle Unicode could have decomposed them as it did with Hangul. This would have greatly reduced the number of required code points, while allowing the display of virtually every conceivable character (which might do away with some of the problems caused by
Han unification Han unification is an effort by the authors of Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Cha ...
). A similar idea is used by some
input method An input method (or input method editor, commonly abbreviated IME) is an operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system softw ...
s, such as
Cangjie ''Cangjie'' () is a legendary figure in ancient China China (), officially the People's Republic of China (PRC; ), is a country in East Asia East Asia is the eastern region of Asia Asia () is Earth's largest and most populo ...
and Wubi. However, attempts to do this for character encoding have stumbled over the fact that Chinese characters do not decompose as simply or as regularly as Hangul does. A set of radicals was provided in Unicode 3.0 (CJK radicals between U+2E80 and U+2EFF, KangXi radicals in U+2F00 to U+2FDF, and ideographic description characters from U+2FF0 to U+2FFB), but the Unicode standard (ch. 12.2 of Unicode 5.2) warns against using
ideographic description sequences The Chinese character description languages are several proposed languages to most accurately and completely describe Chinese (or CJK In internationalization In economics, internationalization or internationalisation is the process of increasin ...
as an alternate representation for previously encoded characters:


Ligatures

Many scripts, including
Arabic Arabic (, ' or , ' or ) is a Semitic language The Semitic languages are a branch of the Afroasiatic language family originating in the Middle East The Middle East is a list of transcontinental countries, transcontinental region ...
and
Devanāgarī Devanagari ( ; , , Sanskrit pronunciation: ), also called Nagari (),Kathleen Kuiper (2010), The Culture of India, New York: The Rosen Publishing Group, , page 83 is a left-to-right abugida An abugida (, from : አቡጊዳ), sometimes ...

Devanāgarī
, have special orthographic rules that require certain combinations of letterforms to be combined into special ligature forms. The rules governing ligature formation can be quite complex, requiring special script-shaping technologies such as ACE (Arabic Calligraphic Engine by DecoType in the 1980s and used to generate all the Arabic examples in the printed editions of the Unicode Standard), which became the
proof of concept Proof of concept (POC), also known as proof of principle, is a realization of a certain method or idea in order to demonstrate its feasibility study, feasibility, or a demonstration in principle with the aim of verifying that some concept or theo ...
for
OpenType OpenType is a format for scalable computer font A computer font (or font) is implemented as a digital data file containing a set of graphically related glyphs, characters, or symbols such as dingbats. Although the term ''font'' first referr ...
(by Adobe and Microsoft),
Graphite Graphite (), archaically referred to as plumbago, is a Crystallinity, crystalline form of the element carbon with its atoms arranged in a Hexagonal crystal system, hexagonal structure. It occurs naturally in this form and is the most stable for ...
(by
SIL International SIL International (formerly known as the Summer Institute of Linguistics) is a evangelical Christian Evangelicalism (), evangelical Christianity, or evangelical Protestantism, is a worldwide trans-denominational movement within Protestan ...
), or AAT (by Apple). Instructions are also embedded in fonts to tell the
operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating systems (OS) like macOS, Linux, Android (operating system), Android and Mi ...

operating system
how to properly output different character sequences. A simple solution to the placement of combining marks or diacritics is assigning the marks a width of zero and placing the glyph itself to the left or right of the left sidebearing (depending on the direction of the script they are intended to be used with). A mark handled this way will appear over whatever character precedes it, but will not adjust its position relative to the width or height of the base glyph; it may be visually awkward and it may overlap some glyphs. Real stacking is impossible, but can be approximated in limited cases (for example, Thai top-combining vowels and tone marks can just be at different heights to start with). Generally this approach is only effective in monospaced fonts, but may be used as a fallback rendering method when more complex methods fail.


Standardized subsets

Several subsets of Unicode are standardized: Microsoft Windows since Windows NT 4.0 supports
WGL-4 Windows Glyph List 4, or more commonly WGL4 for short, also known as the ''Pan-European character set'', is a character Character(s) may refer to: Arts, entertainment, and media Literature * ''Character'' (novel), a 1936 Dutch novel by Ferdin ...
with 657 characters, which is considered to support all contemporary European languages using the Latin, Greek, or Cyrillic script. Other standardized subsets of Unicode include the Multilingual European Subsets: MES-1 (Latin scripts only, 335 characters), MES-2 (Latin, Greek and Cyrillic 1062 characters) and MES-3A & MES-3B (two larger subsets, not shown here). Note that MES-2 includes every character in MES-1 and WGL-4. Rendering software which cannot process a Unicode character appropriately often displays it as an open rectangle, or the Unicode "
replacement character Replace, Replacement or Replacements may refer to: In arts and entertainment Film and television * The Replacement (TV series), ''The Replacement'' (TV series), a 2017 British drama series * The Replacement (Buffy the Vampire Slayer), "The Replace ...

replacement character
" (U+FFFD, �), to indicate the position of the unrecognized character. Some systems have made attempts to provide more information about such characters. Apple's Last Resort font will display a substitute glyph indicating the Unicode range of the character, and the
SIL International SIL International (formerly known as the Summer Institute of Linguistics) is a evangelical Christian Evangelicalism (), evangelical Christianity, or evangelical Protestantism, is a worldwide trans-denominational movement within Protestan ...
's Unicode Fallback font will display a box showing the hexadecimal scalar value of the character.


Mapping and encodings

Several mechanisms have been specified for storing a series of code points as a series of bytes. Unicode defines two mapping methods: the ''Unicode Transformation Format'' (UTF) encodings, and the ''
Universal Coded Character Set The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the International Organization for Standardization, International Standard ISO/International Electrotechnical Commission, IEC  ...
'' (UCS) encodings. An encoding maps (possibly a subset of) the range of Unicode ''code points'' to sequences of values in some fixed-size range, termed ''code units''. All UTF encodings map code points to a unique sequence of bytes. The numbers in the names of the encodings indicate the number of bits per code unit (for UTF encodings) or the number of bytes per code unit (for UCS encodings and
UTF-1 UTF-1 is a method of transforming ISO 10646/Unicode into a stream of bytes. Its design does not provide self-synchronizing code, self-synchronization, which makes searching for Substring, substrings and error recovery difficult. It reuses the ASCII ...
). UTF-8 and UTF-16 are the most commonly used encodings. UCS-2 is an obsolete subset of UTF-16; UCS-4 and UTF-32 are functionally equivalent. UTF encodings include: *
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
, uses one to four bytes for each code point, maximizes compatibility with
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
*
UTF-EBCDIC UTF-EBCDIC is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ), officially t ...
, similar to UTF-8 but designed for compatibility with
EBCDIC Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding six-b ...
(not part of ''The Unicode Standard'') *
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
, uses one or two 16-bit code units per code point, cannot encode surrogates *
UTF-32 UTF-32 (32-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorith ...
, uses one 32-bit code unit per code point UTF-8 uses one to four bytes per code point and, being compact for Latin scripts and ASCII-compatible, provides the ''de facto'' standard encoding for interchange of Unicode text. It is used by
FreeBSD FreeBSD is a free and open-source Free and open-source software (FOSS) is software that is both free software and open-source software where anyone is free software license, freely licensed to use, copy, study, and change the software in any ...

FreeBSD
and most recent
Linux distributions A Linux distribution (often abbreviated as distro) is an operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operati ...
as a direct replacement for legacy encodings in general text handling. The UCS-2 and UTF-16 encodings specify the Unicode Byte Order Mark (BOM) for use at the beginnings of text files, which may be used for byte ordering detection (or byte endianness detection). The BOM, code point U+FEFF, has the important property of unambiguity on byte reorder, regardless of the Unicode encoding used; U+FFFE (the result of byte-swapping U+FEFF) does not equate to a legal character, and U+FEFF in places other than the beginning of text conveys the zero-width non-break space (a character with no appearance and no effect other than preventing the formation of ligatures). The same character converted to UTF-8 becomes the byte sequence EF BB BF. The Unicode Standard allows that the BOM "can serve as signature for UTF-8 encoded text where the character set is unmarked". Some software developers have adopted it for other encodings, including UTF-8, in an attempt to distinguish UTF-8 from local 8-bit
code page In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algorithm) for calculating the greatest common ...
s. However , the UTF-8 standard, recommends that byte order marks be forbidden in protocols using UTF-8, but discusses the cases where this may not be possible. In addition, the large restriction on possible patterns in UTF-8 (for instance there cannot be any lone bytes with the high bit set) means that it should be possible to distinguish UTF-8 from other character encodings without relying on the BOM. In UTF-32 and UCS-4, one
32-bit 32-bit microcomputer A microcomputer is a small, relatively inexpensive computer with a microprocessor as its central processing unit (CPU). It includes a microprocessor, Computer memory, memory and minimal input/output (I/O) circuitry mount ...
code unit serves as a fairly direct representation of any character's code point (although the endianness, which varies across different platforms, affects how the code unit manifests as a byte sequence). In the other encodings, each code point may be represented by a variable number of code units. UTF-32 is widely used as an internal representation of text in programs (as opposed to stored or transmitted text), since every Unix operating system that uses the gcc compilers to generate software uses it as the standard "
wide character A wide character is a computer character (computing), character datatype that generally has a size greater than the traditional 8-bit character. The increased datatype size allows for the use of larger coded Character encoding, character sets. ...
" encoding. Some programming languages, such as
Seed7 Seed7 is an extensible Extensibility is a software engineering and systems design Systems design is the process of defining the architecture File:Plan d'exécution du second étage de l'hôtel de Brionne (dessin) De Cotte 2503c – Galli ...
, use UTF-32 as internal representation for strings and characters. Recent versions of the
Python Python may refer to: * Pythonidae The Pythonidae, commonly known as pythons, are a family of nonvenomous snakes found in Africa, Asia, and Australia. Among its members are some of the largest snakes in the world. Ten genera and 42 species ...
programming language (beginning with 2.2) may also be configured to use UTF-32 as the representation for Unicode strings, effectively disseminating such encoding in
high-level High-level and low-level, as technical terms, are used to classify, describe and point to specific Objective (goal), goals of a systematic operation; and are applied in a wide range of contexts, such as, for instance, in domains as widely varied as ...
coded software.
Punycode Punycode is a representation of Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), ...
, another encoding form, enables the encoding of Unicode strings into the limited character set supported by the
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
-based
Domain Name System The Domain Name System (DNS) is the hierarchical and Decentralised system, decentralized naming system used to identify computers, Internet#Applications_and_services, services, and other resources reachable through the internet or other intern ...

Domain Name System
(DNS). The encoding is used as part of
IDNA Idna ( ar, إذنا) is a Palestinian The Palestinian people ( ar, الشعب الفلسطيني, ''ash-sha‘b al-Filasṭīnī''), also referred to as Palestinians ( ar, الفلسطينيون, links=no, ''al-Filasṭīniyyūn''; he, פָ ...
, which is a system enabling the use of Internationalized Domain Names in all scripts that are supported by Unicode. Earlier and now historical proposals include UTF-5 and UTF-6.
GB18030 GB 18030 is a Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China China (), officially the Peop ...
is another encoding form for Unicode, from the Standardization Administration of China. It is the official character set of the People's Republic of China (PRC). Binary Ordered Compression for Unicode, BOCU-1 and Standard Compression Scheme for Unicode, SCSU are Unicode compression schemes. The April Fools' Day RFC of 2005 specified two parody UTF encodings, UTF-9 and UTF-18.


Adoption


Operating systems

Unicode has become the dominant scheme for internal processing and storage of text. Although a great deal of text is still stored in legacy encodings, Unicode is used almost exclusively for building new information processing systems. Early adopters tended to use UCS-2 (the fixed-width two-byte precursor to UTF-16) and later moved to
UTF-16 UTF-16 (16-bit 16-bit microcomputer A microcomputer is a small, relatively inexpensive computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern computers ...
(the variable-width current standard), as this was the least disruptive way to add support for non-BMP characters. The best known such system is Windows NT (and its descendants, Windows 2000, Windows XP, Windows Vista, Windows 7, Windows 8 and Windows 10), which uses UTF-16 as the sole internal character encoding. The Java virtual machine, Java and .NET Framework, .NET bytecode environments, macOS, and KDE also use it for internal representation. Partial support for Unicode can be installed on Windows 9x through the Microsoft Layer for Unicode.
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
(originally developed for Plan 9 from Bell Labs, Plan 9) has become the main storage encoding on most
Unix-like A Unix-like (sometimes referred to as UN*X or *nix) operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating s ...
operating systems (though others are also used by some libraries) because it is a relatively easy replacement for traditional extended ASCII character sets. UTF-8 is also the most common Unicode encoding used in HTML documents on the
World Wide Web The World Wide Web (WWW), commonly known as the Web, is an information system An information system (IS) is a formal, sociotechnical Sociotechnical systems (STS) in organizational development is an approach to complex organizational ...
. Multilingual text-rendering engines which use Unicode include Uniscribe and DirectWrite for Microsoft Windows, ATSUI and Core Text for macOS, and Pango for GTK+ and the GNOME desktop.


Input methods

Because keyboard layouts cannot have simple key combinations for all characters, several operating systems provide alternative input methods that allow access to the entire repertoire. ISO/IEC 14755, which standardises methods for entering Unicode characters from their code points, specifies several methods. There is the ''Basic method'', where a ''beginning sequence'' is followed by the hexadecimal representation of the code point and the ''ending sequence''. There is also a ''screen-selection entry method'' specified, where the characters are listed in a table in a screen, such as with a character map program. Online tools for finding the code point for a known character include Unicode Lookup by Jonathan Hedley and Shapecatcher by Benjamin Milde. In Unicode Lookup, one enters a search key (e.g. "fractions"), and a list of corresponding characters with their code points is returned. In Shapecatcher, based on Shape context, one draws the character in a box and a list of characters approximating the drawing, with their code points, is returned.


Email

MIME defines two different mechanisms for encoding non-ASCII characters in email, depending on whether the characters are in email headers (such as the "Subject:"), or in the text body of the message; in both cases, the original character set is identified as well as a transfer encoding. For email transmission of Unicode, the
UTF-8 UTF-8 is a variable-width character encoding Character encoding is the process of assigning numbers to Graphics, graphical character (computing), characters, especially the written characters of Language, human language, allowing them to be ...
character set and the Base64 or the Quoted-printable transfer encoding are recommended, depending on whether much of the message consists of
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding Character encoding is the process of assigning numbers to graphical Graphics (from Greek Greek may refer to: Greece Anything of, ...
characters. The details of the two different mechanisms are specified in the MIME standards and generally are hidden from users of email software. The adoption of Unicode in email has been very slow. Some East Asian text is still encoded in encodings such as ISO-2022, and some devices, such as mobile phones, still cannot correctly handle Unicode data. Support has been improving, however. Many major free mail providers such as Yahoo,
Google Google LLC is an American multinational Multinational may refer to: * Multinational corporation, a corporate organization operating in multiple countries * Multinational force, a military body from multiple countries * Multinational stat ...

Google
(Gmail), and
Microsoft Microsoft Corporation is an American multinational corporation, multinational technology company, technology corporation which produces Software, computer software, consumer electronics, personal computers, and related services. Its best-know ...

Microsoft
(Outlook.com) support it.


Web

All W3C recommendations have used Unicode as their ''document character set'' since HTML 4.0. Web browsers have supported Unicode, especially UTF-8, for many years. There used to be display problems resulting primarily from typeface, font related issues; e.g. v 6 and older of Microsoft Internet Explorer did not render many code points unless explicitly told to use a font that contains them. Although syntax rules may affect the order in which characters are allowed to appear,
XML Extensible Markup Language (XML) is a markup language #REDIRECT Markup language In computer text processing, a markup language is a system for annotation, annotating a document in a way that is Syntax (logic), syntactically distinguishable fro ...

XML
(including XHTML) documents, by definition, comprise characters from most of the Unicode code points, with the exception of: * most of the C0 and C1 control codes, C0 control codes, * the permanently unassigned code points D800–DFFF, * FFFE or FFFF. HTML characters manifest either directly as
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
s according to document's encoding, if the encoding supports them, or users may write them as numeric character references based on the character's Unicode code point. For example, the references Δ, Й, ק, م, ๗, あ, 叶, 葉, and 말 (or the same numeric values expressed in hexadecimal, with &#x as the prefix) should display on all browsers as Δ, Й, ק ,م, ๗, あ, 叶, 葉, and 말. When specifying Uniform Resource Identifier, URIs, for example as URLs in HTTP requests, non-ASCII characters must be percent encoding, percent-encoded.


Fonts

Unicode is not in principle concerned with fonts ''per se'', seeing them as implementation choices. Any given character may have many allographs, from the more common bold, italic and base letterforms to complex decorative styles. A font is "Unicode compliant" if the glyphs in the font can be accessed using code points defined in the Unicode standard. The standard does not specify a minimum number of characters that must be included in the font; some fonts have quite a small repertoire. Free and retail
font In metal A metal (from Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ), officially the Hellenic Republic, is a country located in Southeast Europe. Its population is approxim ...

font
s based on Unicode are widely available, since TrueType and
OpenType OpenType is a format for scalable computer font A computer font (or font) is implemented as a digital data file containing a set of graphically related glyphs, characters, or symbols such as dingbats. Although the term ''font'' first referr ...
support Unicode. These font formats map Unicode code points to glyphs, but TrueType font is restricted to 65,535 glyphs. List of typefaces, Thousands of fonts exist on the market, but fewer than a dozen fonts—sometimes described as "pan-Unicode" fonts—attempt to support the majority of Unicode's character repertoire. Instead, Unicode-based List of Unicode fonts, fonts typically focus on supporting only basic ASCII and particular scripts or sets of characters or symbols. Several reasons justify this approach: applications and documents rarely need to render characters from more than one or two writing systems; fonts tend to demand resources in computing environments; and operating systems and applications show increasing intelligence in regard to obtaining glyph information from separate font files as needed, i.e., font substitution. Furthermore, designing a consistent set of rendering instructions for tens of thousands of glyphs constitutes a monumental task; such a venture passes the point of diminishing returns for most typefaces.


Newlines

Unicode partially addresses the newline problem that occurs when trying to read a text file on different platforms. Unicode defines a large number of Newline#Unicode, characters that conforming applications should recognize as line terminators. In terms of the newline, Unicode introduced and . This was an attempt to provide a Unicode solution to encoding paragraphs and lines semantically, potentially replacing all of the various platform solutions. In doing so, Unicode does provide a way around the historical platform dependent solutions. Nonetheless, few if any Unicode solutions have adopted these Unicode line and paragraph separators as the sole canonical line ending characters. However, a common approach to solving this issue is through newline normalization. This is achieved with the Cocoa text system in Mac OS X and also with W3C XML and HTML recommendations. In this approach every possible newline character is converted internally to a common newline (which one does not really matter since it is an internal operation just for rendering). In other words, the text system can correctly treat the character as a newline, regardless of the input's actual encoding.


Issues


Philosophical and completeness criticisms

Han unification Han unification is an effort by the authors of Unicode Unicode, formally the Unicode Standard, is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Cha ...
(the identification of forms in the East Asian languages which one can treat as stylistic variations of the same historical character) has become one of the most controversial aspects of Unicode, despite the presence of a majority of experts from all three regions in the Ideographic Research Group (IRG), which advises the Consortium and ISO on additions to the repertoire and on Han unification. Unicode has been criticized for failing to separately encode older and alternative forms of kanji which, critics argue, complicates the processing of ancient Japanese and uncommon Japanese names. This is often due to the fact that Unicode encodes characters rather than glyphs (the visual representations of the basic character that often vary from one language to another). Unification of glyphs leads to the perception that the languages themselves, not just the basic character representation, are being merged.The secret life of Unicode: A peek at Unicode's soft underbelly
Suzanne Topping, 1 May 2001 ''(Internet Archive)''
There have been several attempts to create alternative encodings that preserve the stylistic differences between Chinese, Japanese, and Korean characters in opposition to Unicode's policy of Han unification. An example of one is TRON (encoding), TRON (although it is not widely adopted in Japan, there are some users who need to handle historical Japanese text and favor it). Although the repertoire of fewer than 21,000 Han characters in the earliest version of Unicode was largely limited to characters in common modern usage, Unicode now includes more than 92,000 Han characters, and work is continuing to add thousands more historic and dialectal characters used in China, Japan, Korea, Taiwan, and Vietnam. Modern font technology provides a means to address the practical issue of needing to depict a unified Han character in terms of a collection of alternative glyph representations, in the form of variation Selectors, Unicode variation sequences. For example, the Advanced Typographic tables of
OpenType OpenType is a format for scalable computer font A computer font (or font) is implemented as a digital data file containing a set of graphically related glyphs, characters, or symbols such as dingbats. Although the term ''font'' first referr ...
permit one of a number of alternative glyph representations to be selected when performing the character to glyph mapping process. In this case, information can be provided within plain text to designate which alternate character form to select. If the difference in the appropriate glyphs for two characters in the same script differ only in the italic, Unicode has generally unified them, as can be seen in the comparison between Russian (labeled standard) and Serbian characters at right, meaning that the differences are displayed through smart font technology or manually changing fonts.


Mapping to legacy character sets

Unicode was designed to provide code-point-by-code-point round-trip format conversion to and from any preexisting character encodings, so that text files in older character sets can be converted to Unicode and then back and get back the same file, without employing context-dependent interpretation. That has meant that inconsistent legacy architectures, such as combining character, combining diacritics and precomposed characters, both exist in Unicode, giving more than one method of representing some text. This is most pronounced in the three different encoding forms for Korean
Hangul The Korean alphabet, known as Hangul, . Hangul may also be written as following South Korea's standard Romanization. in South Korea South Korea, officially the Republic of Korea (ROK), is a country in East Asia, constituting the ...

Hangul
. Since version 3.0, any precomposed characters that can be represented by a combining sequence of already existing characters can no longer be added to the standard in order to preserve interoperability between software using different versions of Unicode. Injective mappings must be provided between characters in existing legacy character sets and characters in Unicode to facilitate conversion to Unicode and allow interoperability with legacy software. Lack of consistency in various mappings between earlier Japanese encodings such as Shift-JIS or EUC-JP and Unicode led to round-trip format conversion mismatches, particularly the mapping of the character JIS X 0208 '~' (1-33, WAVE DASH), heavily used in legacy database data, to either (in Microsoft Windows) or (other vendors). Some Japanese computer programmers objected to Unicode because it requires them to separate the use of and , which was mapped to 0x5C in JIS X 0201, and a lot of legacy code exists with this usage. (This encoding also replaces tilde '~' 0x7E with macron '¯', now 0xAF.) The separation of these characters exists in ISO 8859-1, from long before Unicode.


Indic scripts

Indic scripts such as Tamil script, Tamil and
Devanagari Devanagari ( ; , , Sanskrit pronunciation: ), also called Nagari (),Kathleen Kuiper (2010), The Culture of India, New York: The Rosen Publishing Group, , page 83 is a left-to-right abugida, based on the ancient Brahmi script, ''Brāhmī'' sc ...

Devanagari
are each allocated only 128 code points, matching the ISCII standard. The correct rendering of Unicode Indic text requires transforming the stored logical order characters into visual order and the forming of ligatures (aka conjuncts) out of components. Some local scholars argued in favor of assignments of Unicode code points to these ligatures, going against the practice for other writing systems, though Unicode contains some Arabic and other ligatures for backward compatibility purposes only. Encoding of any new ligatures in Unicode will not happen, in part because the set of ligatures is font-dependent, and Unicode is an encoding independent of font variations. The same kind of issue arose for the Tibetan script in 2003 when the Standardization Administration of China proposed encoding 956 precomposed Tibetan syllables, but these were rejected for encoding by the relevant ISO committee (ISO/IEC JTC 1/SC 2). Thai alphabet support has been criticized for its ordering of Thai characters. The vowels เ, แ, โ, ใ, ไ that are written to the left of the preceding consonant are in visual order instead of phonetic order, unlike the Unicode representations of other Indic scripts. This complication is due to Unicode inheriting the TIS-620, Thai Industrial Standard 620, which worked in the same way, and was the way in which Thai had always been written on keyboards. This ordering problem complicates the Unicode collation process slightly, requiring table lookups to reorder Thai characters for collation. Even if Unicode had adopted encoding according to spoken order, it would still be problematic to collate words in dictionary order. E.g., the word "perform" starts with a consonant cluster "สด" (with an inherent vowel for the consonant "ส"), the vowel แ-, in spoken order would come after the ด, but in a dictionary, the word is collated as it is written, with the vowel following the ส.


Combining characters

Characters with diacritical marks can generally be represented either as a single precomposed character or as a decomposed sequence of a base letter plus one or more non-spacing marks. For example, ḗ (precomposed e with macron and acute above) and ḗ (e followed by the combining macron above and combining acute above) should be rendered identically, both appearing as an e with a Macron (diacritic), macron and
acute accent The acute accent, , is a diacritic A diacritic (also diacritical mark, diacritical point, diacritical sign, or accent) is a glyph The term glyph is used in typography File:metal movable type.jpg, 225px, Movable type being assembl ...

acute accent
, but in practice, their appearance may vary depending upon what rendering engine and fonts are being used to display the characters. Similarly, dot (diacritic), underdots, as needed in the romanization of Indo-Aryan languages, Indic, will often be placed incorrectly.. Unicode characters that map to precomposed glyphs can be used in many cases, thus avoiding the problem, but where no precomposed character has been encoded the problem can often be solved by using a specialist Unicode font such as Charis SIL that uses
Graphite Graphite (), archaically referred to as plumbago, is a Crystallinity, crystalline form of the element carbon with its atoms arranged in a Hexagonal crystal system, hexagonal structure. It occurs naturally in this form and is the most stable for ...
,
OpenType OpenType is a format for scalable computer font A computer font (or font) is implemented as a digital data file containing a set of graphically related glyphs, characters, or symbols such as dingbats. Although the term ''font'' first referr ...
, or AAT technologies for advanced rendering features.


Anomalies

The Unicode standard has imposed rules intended to guarantee stability. Depending on the strictness of a rule, a change can be prohibited or allowed. For example, a "name" given to a code point cannot and will not change. But a "script" property is more flexible, by Unicode's own rules. In version 2.0, Unicode changed many code point "names" from version 1. At the same moment, Unicode stated that from then on, an assigned name to a code point would never change anymore. This implies that when mistakes are published, these mistakes cannot be corrected, even if they are trivial (as happened in one instance with the spelling for in a character name). In 2006 a list of anomalies in character names was first published, and, as of June 2021, there were 104 characters with identified issues, for example: * : This is a small letter. The capital is . * : Does not join graphemes. * : This is not a Yi syllable, but a Yi iteration mark. * : ''bracket'' is spelled incorrectly. Spelling errors are resolved by using Unicode alias names and abbreviations.


Security issues

Unicode has a large number of homoglyphs, many of which look very similar or identical to ASCII letters. Substitution of these can make an identifier or URL that looks correct, but directs to a different location than expected. Mitigation requires disallowing these characters, displaying them differently, or requiring that they resolve to the same identifier, all of this is complicated due to the huge and constantly changing set of characters. A security advisory has been recently released from two researchers, one from the University of Cambridge and the other from the same and from the University of Edinburgh, in which they assert that the Bidirectional Text, BIDI codes can be used to make large sections of code do something different from what they appear to do. Some security groups (like the one for Rust language) are already taking measures and issuing their own security advisories.Security advisory for rustc (CVE-2021-42574)
/ref>


See also

* Comparison of Unicode encodings * Religious and political symbols in Unicode * International Components for Unicode (ICU), now as ICU-TC a part of Unicode * List of binary codes * List of Unicode characters * List of XML and HTML character entity references * Open-source Unicode typefaces * Standards related to Unicode * Unicode symbols *
Universal Coded Character Set The Universal Coded Character Set (UCS, Unicode) is a standard set of character (computing), characters defined by the International Organization for Standardization, International Standard ISO/International Electrotechnical Commission, IEC  ...
* Lotus Multi-Byte Character Set (LMBCS), a parallel development with similar intentions


Notes


References


Further reading

* ''The Unicode Standard, Version 3.0'', The Unicode Consortium, Addison-Wesley Longman, Inc., April 2000. * ''The Unicode Standard, Version 4.0'', The Unicode Consortium, Addison-Wesley Professional, 27 August 2003. * ''The Unicode Standard, Version 5.0, Fifth Edition'', The
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) A 501(c)(3) organization is a corporation, trust, unincorporated association, or other type of organization exempt from federal income tax under section 501(c)(3) of Title 26 of the ...
, Addison-Wesley Professional, 27 October 2006. * Julie D. Allen. ''The Unicode Standard, Version 6.0'', The
Unicode Consortium The Unicode Consortium (legally Unicode, Inc.) is a 501(c)(3) A 501(c)(3) organization is a corporation, trust, unincorporated association, or other type of organization exempt from federal income tax under section 501(c)(3) of Title 26 of the ...
, Mountain View, 2011, ,

. * ''The Complete Manual of Typography'', James Felici, Adobe Press; 1st edition, 2002. * ''Unicode: A Primer'', Tony Graham, M&T books, 2000. . * ''Unicode Demystified: A Practical Programmer's Guide to the Encoding Standard'', Richard Gillam, Addison-Wesley Professional; 1st edition, 2002. * ''Unicode Explained'', Jukka K. Korpela, O'Reilly; 1st edition, 2006. *


External links

* * Th
Unicode Character Database

text document
listing the names, code points and properties of all Unicode characters *
Alan Wood's Unicode Resources
contains lists of word processors with Unicode capability; fonts and characters are grouped by type; characters are presented in lists, not grids.
Unicode BMP Fallback Font
displays the Unicode 6.1 value of any character in a document, including in the Private Use Area, rather than the glyph itself. {{Authority control Unicode, Character encoding Digital typography