HOME

TheInfoList




Character encoding is the process of assigning numbers to
graphical Graphics (from Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ), officially the Hellenic Republic, is a country located in Southeast Europe. Its population is approximately 10.7 millio ...

graphical
characters Character(s) may refer to: Arts, entertainment, and media Literature * Character (novel), ''Character'' (novel), a 1936 Dutch novel by Ferdinand Bordewijk * Characters (Theophrastus), ''Characters'' (Theophrastus), a classical Greek set of char ...
, especially the written characters of
human language A language is a structured system of communication used by humans, based on speech and gesture (spoken language), Signed language, sign, or often Written language, writing. The structure of language is its grammar and the free components are i ...

human language
, allowing them to be stored, transmitted, and
transformed
transformed
using
digital Digital usually refers to something using digits, particularly binary digits. Technology and computing Hardware *Digital electronics Digital electronics is a field of electronics Electronics comprises the physics, engineering, technology a ...
computers A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations automatically. Modern computers can perform generic sets of operations known as Computer program, programs. These p ...

computers
. The numerical values that make up a character encoding are known as "
code point In character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardw ...
s" and collectively comprise a "code space", a "
code page In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algorithm) for calculating the greatest common ...
", or a " character map". Early character codes associated with the optical or electrical
telegraph Telegraphy is the long-distance transmission of messages where the sender uses symbolic codes, known to the recipient, rather than a physical exchange of an object bearing the message. Thus is a method of telegraphy, whereas is not. Ancien ...
could only represent a subset of the characters used in
written language A written language is the representation of a spoken or gestural language A language is a structured system of communication used by humans, including speech (spoken language), gestures (Signed language, sign language) and writing. Most la ...
s, sometimes restricted to upper case letters,
numeral A numeral is a figure, symbol, or group of figures or symbols denoting a number. It may refer to: * Numeral system used in mathematics * Numeral (linguistics), a part of speech denoting numbers (e.g. ''one'' and ''first'' in English) * Numerical di ...
s and some
punctuation Punctuation (or sometimes interpunction) is the use of spacing, conventional signs (called punctuation marks), and certain typographical devices as aids to the understanding and correct reading of written text, whether read silently or aloud. Ano ...
only. The low cost of digital representation of data in modern computer systems allows more elaborate character codes (such as
Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
) which represent most of the characters used in many written languages. Character encoding using internationally accepted standards permits worldwide interchange of text in electronic form.


History

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. The earliest codes were based upon manual and hand-written encoding and cyphering systems, such as
Bacon's cipher Bacon's cipher or the Baconian cipher is a method of steganography, steganographic message encoding devised by Francis Bacon in 1605. A message is concealed in the presentation of text, rather than its content. Cipher details To encode a message, e ...
,
Braille Braille ( ; Braille: ⠃⠗⠇; ) is a tactile writing system A writing system is a method of visually representing verbal communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, ...

Braille
,
International maritime signal flags International maritime signal flags are various flags used to communicate with ships. The principal system of flags and associated codes is the International Code of Signals. Various navies have flag systems with additional flags and codes, and o ...
, and the 4-digit encoding of Chinese characters for a
Chinese telegraph code The Chinese telegraph code, Chinese telegraphic code, or Chinese commercial code ( or ) is a four-digit decimal code ( character encoding) for electrically telegraphing messages written with Chinese character Chinese characters, also cal ...
(
Hans Schjellerup Hans Carl Frederik Christian Schjellerup (8 February 1827 – 13 November 1887) was a Danish astronomer. He was born at Odense, the son of a jeweller. Initially he was apprenticed as a watch maker, but in 1848 he passed the entrance exam for th ...

Hans Schjellerup
, 1869). With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. The earliest well-known electrically-transmitted character code,
Morse code Morse code is a method used in telecommunication to Character encoding, encode Written language, text characters as standardized sequences of two different signal durations, called ''dots'' and ''dashes'', or ''dits'' and ''dahs''. Morse code ...
, introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Though some commercial use of Morse code was via machinery, it was often used as a manual code, generated by hand on a
telegraph key A telegraph key is a specialized electrical switch used by a trained operator to transmit text messages in Morse code in a telegraphy system. Keys are used in all forms of electrical telegraph systems, including landline (also called wire) te ...
and decipherable by ear, and persists in
amateur radio Amateur radio, also known as ham radio, is the use of radio frequency radio spectrum, spectrum for purposes of non-commercial exchange of messages, wireless experimentation, self-training, private recreation, radiosport, contesting, and emerge ...
and
aeronautical Aeronautics is the science or art involved with the study, design, and manufacturing of air flight Flight or flying is the process by which an object (physics), object motion (physics), moves through a space without contacting any planetary ...
use. Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g.
Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
). Common examples of character encoding systems include
Morse code Morse code is a method used in telecommunication to Character encoding, encode Written language, text characters as standardized sequences of two different signal durations, called ''dots'' and ''dashes'', or ''dits'' and ''dahs''. Morse code ...
, the
Baudot code The Baudot code is an early character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and developme ...
, the American Standard Code for Information Interchange (
ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the stu ...
) and
Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
.
Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
, a well defined and extensible encoding system, has supplanted most earlier character encodings, but the path of code development to the present is fairly well known. The
Baudot code The Baudot code is an early character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and developme ...
, a five-bit encoding, was created by
Émile Baudot Jean-Maurice-Émile Baudot (; 11 September 1845 – 28 March 1903), French telegraph Telegraphy is the long-distance transmission of textual messages where the sender uses a semaphore system, known to the recipient, rather than a physica ...
in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930. The name "baudot" has been erroneously applied to ITA2 and its many variants. ITA2 suffered from many shortcomings and was often "improved" by many equipment manufacturers, sometimes creating compatibility issues. In 1959 the U.S. military defined its
Fieldata
Fieldata
code, a six-or seven-bit code, introduced by the U.S. Army Signal Corps. While Fieldata addressed many of the then-modern issues (e.g. letter and digit codes arranged for machine collation), Fieldata fell short of its goals and was short-lived. In 1963 the first ASCII (American Standard Code for Information Interchange) code was released (X3.4-1963) by the ASCII committee (which contained at least one member of the Fieldata committee, W. F. Leubbert) which addressed most of the shortcomings of Fieldata, using a simpler code. Many of the changes were subtle, such as collatable character sets within certain numeric ranges. ASCII63 was a success, widely adopted by industry, and with the follow-up issue of the 1967 ASCII code (which added lower-case letters and fixed some "control code" issues) ASCII67 was adopted fairly widely. ASCII67's American-centric nature was somewhat addressed in the European
ECMA-6 ISO/IEC 646 is the name of a set of ISO standards, described as ''Information technology — ISO 7-bit coded character set for information interchange'' and developed in cooperation with ASCII at least since 1964. Since its first edi ...
standard.
Herman Hollerith Herman Hollerith (February 29, 1860 – November 17, 1929) was an American businessman, inventor, and statistician who developed an electromechanical tabulating machine The tabulating machine was an electromechanical machine designed to assist ...
invented punch card data encoding in the late 19th century to analyze census data. Initially, each hole position represented a different data element, but later, numeric information was encoded by numbering the lower rows 0 to 9, with a punch in a column representing its row number. Later alphabetic data was encoded by allowing more than one punch per column. Electromechanical
tabulating machine The tabulating machine was an electromechanical In engineering Engineering is the use of scientific method, scientific principles to design and build machines, structures, and other items, including bridges, tunnels, roads, vehicles, ...
s represented date internally by the timing of pulses relative to the motion of the cards through the machine. When
IBM International Business Machines Corporation (IBM) is an American multinational technology company headquartered in Armonk, New York, with operations in over 170 countries. The company began in 1911, founded in Endicott, New York, as the C ...

IBM
went to electronic processing, starting with the
IBM 603 The IBM 603 Electronic Multiplier was the first mass-produced commercial electronic calculating device; it used full-size vacuum tube A vacuum tube, an electron tube, valve (British usage) or tube (North America), is a device that controls ele ...
Electronic Multiplier, it used a variety of binary encoding schemes that were tied to the punch card code.
IBM International Business Machines Corporation (IBM) is an American multinational technology company headquartered in Armonk, New York, with operations in over 170 countries. The company began in 1911, founded in Endicott, New York, as the C ...

IBM
's
Binary Coded Decimal In computing and electronics, electronic systems, binary-coded decimal (BCD) is a class of Binary numeral system, binary encodings of decimal numbers where each numerical digit, digit is represented by a fixed number of bits, usually four or eig ...
( BCD) was a six-bit encoding scheme used by IBM as early as 1953 in its 702 and
704 __NOTOC__ Year 704 (Roman numerals, DCCIV) was a leap year starting on Tuesday (link will display the full calendar) of the Julian calendar. The denomination 704 for this year has been used since the early medieval period, when the Anno Domini c ...
computers, and in its later
7000 Series7000 series may refer to: Japanese trains * Chichibu Railway 7000 series The is an electric multiple unit (EMU) train type for local services on the Chichibu Main Line operated by the private railway operator Chichibu Railway in Japan since Mar ...
and
1400 series Fourteen or 14 may refer to: * 14 (number), the natural number following 13 and preceding 15 * one of the years 14 BC, AD 14, 1914, 2014 Music * 14th (band), a British electronic music duo * 14 (David Garrett album), ''14'' (David Garrett album), ...
, as well as in associated peripherals. Since the punched card code then in use only allowed digits, upper-case English letters and a few special characters, 6 bits were sufficient. BCD extended existing simple four-bit numeric encoding to include alphabetic and special characters, mapping it easily to punch-card encoding which was already in widespread use. IBMs codes were used primarily with IBM equipment; other computer vendors of the era had their own character codes, often 6 bit, but usually had the ability to read tapes produced on IBM equipment. BCD was the precursor
IBM International Business Machines Corporation (IBM) is an American multinational technology company headquartered in Armonk, New York, with operations in over 170 countries. The company began in 1911, founded in Endicott, New York, as the C ...

IBM
's
Extended Binary Coded Decimal Interchange Code Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes t ...
(usually abbreviated as EBCDIC), an eight-bit encoding scheme developed in 1963 for the
IBM System/360 The IBM System/360 (S/360) is a family of mainframe computer systems that was announced by IBM on April 7, 1964, and delivered between 1965 and 1978. It was the first family of computers designed to cover the complete range of applications, fr ...
that featured a larger character set, including lower case letters. The limitations of such sets soon became apparent, and a number of ''ad hoc'' methods were developed to extend them. The need to support more
writing system A writing system is a method of visually representing verbal communication Communication (from Latin ''communicare'', meaning "to share" or "to be in relation with") is "an apparent answer to the painful divisions between self and other, p ...
s for different languages, including the
CJK In internationalization, CJK characters is a collective term for the Chinese, Japanese, and Korean Korean may refer to: People and culture * Koreans, an ethnic group originating in the Korean Peninsula * Korean cuisine * Korean culture * Kor ...
family of East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ''ad hoc'' approaches. In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that, on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). In 1985, the average personal computer user's
hard disk drive A hard disk drive (HDD), hard disk, hard drive, or fixed disk is an electro-mechanical data storage device File:Reel-to-reel recorder tc-630.jpg, On a reel-to-reel tape recorder (Sony TC-630), the recorder is data storage equipment an ...

hard disk drive
could store only about 10 megabytes, and it cost approximately US$250 on the wholesale market (and much higher if purchased separately at retail), so it was very important at the time to make every bit count. The compromise solution that was eventually found and developed into Unicode was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers called
code point In character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardw ...
s. Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. To encode code points higher than the length of the code unit, such as above 256 for 8-bit units, the solution was to implement
variable-width encoding A variable-width encoding is a type of character encoding scheme in which codes of differing lengths are used to encode a character set (a repertoire of symbols) for representation in a computer A computer is a machine that can be program ...
s where an escape sequence would signal that subsequent bits should be parsed as a higher code point.


Terminology

; Terminology related to character encoding: * A ''character'' is a minimal unit of text that has semantic value. * A ''character set'' is a collection of characters that might be used by multiple languages. ''Example:'' The Latin character set is used by English and most European languages, though the Greek character set is used only by the Greek language. * A ''coded character set'' is a character set in which each character corresponds to a unique number. * A ''code point'' of a coded character set is any allowed value in the character set or code space. * A ''code space'' is a range of integers whose values are code points * A ''code unit'' is a bit sequence used to encode each character of a repertoire within a given encoding form. This is referred to as a ''code value'' in some documents. ; Character repertoire (the abstract set of characters): The character repertoire is an abstract set of more than one million characters found in a wide variety of scripts including Latin, Cyrillic, Chinese, Korean, Japanese, Hebrew, and Aramaic. Other symbols such as musical notation are also included in the character repertoire. Both the Unicode and
GB18030 GB 18030 is a Guobiao standards, Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China. GB18030 is the regi ...
standards have a character repertoire. As new characters are added to one standard, the other standard also adds those characters, to maintain parity. The code unit size is equivalent to the bit measurement for the particular encoding: * A code unit in
US-ASCII ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the stud ...
consists of 7 bits; * A code unit in
UTF-8 UTF-8 is a variable-width character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development ...
,
EBCDIC Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes t ...
and GB18030 consists of 8 bits; * A code unit in
UTF-16 UTF-16 ( Transformation Format) is a capable of encoding all 1,112,064 valid character s of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is , as code points are encoded with one or two 16-bit ''c ...
consists of 16 bits; * A code unit in
UTF-32 UTF-32 (32-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and deve ...
consists of 32 bits. ''Example of a code unit:'' Consider a
string String or strings may refer to: *String (structure), a long flexible structure made from threads twisted together, which is used to tie, bind, or hang other objects Arts, entertainment, and media Films * Strings (1991 film), ''Strings'' (1991 fil ...
of the letters "abc" followed by (represented with 1 char32_t, 2 char16_t or 4 char8_t). That string contains: * four characters; * four code points * either: *: four code units in UTF-32 (00000061, 00000062, 00000063, 00010400) *: five code units in UTF-16 (0061, 0062, 0063, d801, dc00), or *: seven code units in UTF-8 (61, 62, 63, f0, 90, 90, 80). The convention to refer to a character in Unicode is to start with 'U+' followed by the codepoint value in hexadecimal. The range of valid code points for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17 planes, identified by the numbers 0 to 16. Characters in the range U+0000 to U+FFFF are in plane 0, called the
Basic Multilingual Plane In the Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, conventi ...
(BMP). This plane contains most commonly-used characters. Characters in the range U+10000 to U+10FFFF in the other planes are called supplementary characters. The following table shows examples of code point values: A code point is represented by a sequence of code units. The mapping is defined by the encoding. Thus, the number of code units required to represent a code point depends on the encoding: * ''UTF-8:'' code points map to a sequence of one, two, three or four code units. * ''UTF-16:'' code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16: "Unicode surrogate pairs". * ''UTF-32:'' the 32-bit code unit is large enough that every code point is represented as a single code unit. * ''GB18030:'' multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.


Unicode encoding model

Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
and its parallel standard, the ISO/IEC 10646
Universal Character Set The Universal Coded Character Set (UCS, Unicode) is a standard set of characters Character(s) may refer to: Arts, entertainment, and media Literature * Character (novel), ''Character'' (novel), a 1936 Dutch novel by Ferdinand Bordewijk * Ch ...
, together constitute a modern, unified character encoding. Rather than mapping characters directly to octets (
byte The byte is a unit of digital information that most commonly consists of eight bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It ...
s), they separately define what characters are available, corresponding natural numbers (
code point In character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardw ...
s), how those numbers are encoded as a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets. The purpose of this decomposition is to establish a universal set of characters that can be encoded in a variety of ways. To describe this model correctly requires more precise terms than "character set" and "character encoding." The terms used in the modern model follow: A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, i.e. no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the
Windows code pageWindows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode in Microsoft Windows, Uni ...
s). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into basic information units. The basic variants of the
Latin Latin (, or , ) is a classical language belonging to the Italic branch of the Indo-European languages. Latin was originally spoken in the area around Rome, known as Latium. Through the power of the Roman Republic, it became the dominant la ...

Latin
,
Greek#REDIRECT Greek Greek may refer to: Greece Anything of, from, or related to Greece Greece ( el, Ελλάδα, , ), officially the Hellenic Republic, is a country located in Southeast Europe. Its population is approximately 10.7 million as of ...

Greek
and
Cyrillic , bg, кирилица , mk, кирилица , russian: кириллица , sr, ћирилица, uk, кирилиця , fam1 = Egyptian hieroglyphs , fam2 = Proto-Sinaitic , fam3 = Phoenician alphabet, Phoenician , ...
alphabets can be broken down into letters, digits, punctuation, and a few ''special characters'' such as the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. But even with these alphabets,
diacritic A diacritic (also diacritical mark, diacritical point, diacritical sign, or accent) is a glyph added to a letter Letter, letters, or literature may refer to: Characters typeface * Letter (alphabet) A letter is a segmental symbol A sy ...
s pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Ligatures pose similar problems. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and
glyph The term glyph is used in typography File:metal movable type.jpg, 225px, Movable type being assembled on a composing stick using pieces that are stored in the type case shown below it Typography is the art and technique of typesetting, arr ...
s that are joined together in different ways for different situations. A coded character set (CCS) is a
function Function or functionality may refer to: Computing * Function key A function key is a key on a computer A computer is a machine that can be programmed to carry out sequences of arithmetic or logical operations automatically. Modern comp ...
that maps characters to ''
code point In character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardw ...
s'' (each code point represents one character). For example, in a given repertoire, the capital letter "A" in the Latin alphabet might be represented by the code point 65, the character "B" to 66, and so on. Multiple coded character sets may share the same repertoire; for example
ISO/IEC 8859-1 ISO/IEC 8859-1:1998, ''Information technology — 8-bit In computer architecture In computer engineering, computer architecture is a set of rules and methods that describe the functionality, organization, and implementation of compu ...
and IBM code pages 037 and 500 all cover the same repertoire but map them to different code points. A character encoding form (CEF) is the mapping of code points to ''code units'' to facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. This correspondence is defined by a CEF. Next, a character encoding scheme (CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Simple character encoding schemes include
UTF-8 UTF-8 is a variable-width character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development ...
,
UTF-16BE UTF-16 (16-bit computing, 16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid character code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding ...
, UTF-32BE, UTF-16LE or UTF-32LE; compound character encoding schemes, such as
UTF-16 UTF-16 ( Transformation Format) is a capable of encoding all 1,112,064 valid character s of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is , as code points are encoded with one or two 16-bit ''c ...
,
UTF-32 UTF-32 (32-bit The bit is a basic unit of information in computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and deve ...
and
ISO/IEC 2022 ISO/IEC 2022 ''Information technology—Character code structure and extension techniques'', is an ISO The International Organization for Standardization (ISO; ) is an international standard are technical standards developed by internationa ...
, switch between several simple schemes by using byte order marks or
escape sequence In computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of Algo ...
s; compressing schemes try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and
PunycodePunycode is a representation of Unicode Unicode is an information technology Technical standard, standard for the consistent character encoding, encoding, representation, and handling of Character (computing), text expressed in most of the world ...
). Although UTF-32BE is a simpler CES, most systems working with Unicode use either
UTF-8 UTF-8 is a variable-width character encoding In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development ...
, which is
backward compatible Backward or Backwards is a relative direction Body relative directions (also known as egocentric coordinates) are geometrical orientations relative to a body such as a human Humans (''Homo sapiens'') are the most populous and widespre ...
with fixed-width ASCII and maps Unicode code points to variable-width sequences of octets, or
UTF-16BE UTF-16 (16-bit computing, 16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid character code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding ...
, which is
backward compatible Backward or Backwards is a relative direction Body relative directions (also known as egocentric coordinates) are geometrical orientations relative to a body such as a human Humans (''Homo sapiens'') are the most populous and widespre ...
with fixed-width UCS-2BE and maps Unicode code points to variable-width sequences of 16-bit words. See
comparison of Unicode encodings This article compares Unicode encodings. Two situations are considered: 8-bit-clean environments (which can be assumed), and environments that forbid use of byte values that have the high bit set. Originally such prohibitions were to allow for link ...
for a detailed discussion. Finally, there may be a higher level protocol which supplies additional information to select the particular variant of a
Unicode Unicode, formally the Unicode Standard, is an information technology standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requiremen ...

Unicode
character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the
XML Extensible Markup Language (XML) is a markup language #REDIRECT Markup language In computer text processing, a markup language is a system for annotation, annotating a document in a way that is Syntax (logic), syntactically distinguishable fro ...

XML
attribute xml:lang. The Unicode model uses the term character map for historical systems which directly assign a sequence of characters to a sequence of bytes, covering all of CCS, CEF and CES layers.


Character sets, character maps and code pages

Historically, the terms "character encoding", "character map", "character set" and "
code page In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algorithm) for calculating the greatest common ...
" were synonymous in
computer science Computer science deals with the theoretical foundations of information, algorithms and the architectures of its computation as well as practical techniques for their application. Computer science is the study of , , and . Computer science ...
, as the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units – usually with a single character per code unit. But now the terms have related but distinct meanings, due to efforts by standards bodies to use precise terminology when writing about and unifying many different encoding systems. Regardless, the terms are still used interchangeably, with ''character set'' being nearly ubiquitous. A "code page" usually means a
byte-oriented Byte-oriented framing protocol is "a communications protocol in which full bytes are used as control codes. Also known as character-oriented protocol." For example UART communication is byte-oriented. The term "character-oriented" is deprecated, ...
encoding, but with regard to some suite of encodings (covering different scripts), where many characters share the same
codes In communications and information processing, code is a system of rules to convert information—such as a letter (alphabet), letter, word, sound, image, or gesture—into another form, sometimes data compression, shortened or secrecy, secret, f ...
in most or all those code pages. Well-known code page suites are "Windows" (based on Windows-1252) and "IBM"/"DOS" (based on
code page 437 Code page 437 (CCSID A CCSID (coded character set identifier) is a 16-bit number that represents a particular encoding In communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, mea ...
), see
Windows code pageWindows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode in Microsoft Windows, Uni ...
for details. Most, but not all, encodings referred to as code pages are single-byte encodings (but see
octet Octet may refer to: Music * Octet (music) In music, an octet is a musical ensemble consisting of eight Musical instrument, instruments or voices, or a musical composition written for such an ensemble. Octets in classical music Octets in Europea ...
on byte size.) IBM's Character Data Representation Architecture (CDRA) designates entities with coded character set identifiers (
CCSID A CCSID (coded character set identifier) is a 16-bit number that represents a particular encoding In communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, meaning among Subject (phil ...
s), each of which is variously called a "charset", "character set", "code page", or "CHARMAP". The term "code page" does not occur in Unix or Linux where "charmap" is preferred, usually in the larger context of locales. In contrast to a "
coded character set In computing, Computer data storage, data storage, and data transmission, character encoding is used to represent a repertoire of character (computing), characters by some kind of code, encoding system that assigns a number to each character for ...
", a "character encoding" is a map from abstract characters to
code word In communication Communication (from Latin ''communicare'', meaning "to share" or "to be in relation with") is "an apparent answer to the painful divisions between self and other, private and public, and inner thought and outer world." As this ...
s. A "character set" in
HTTP The Hypertext Transfer Protocol (HTTP) is an application layer An application layer is an abstraction layer that specifies the shared communications protocols and Interface (computing), interface methods used by Host (network), hosts in a c ...

HTTP
(and
MIME #REDIRECT Mime artist A mime artist or just mime (from Greek , , "imitator, actor") is a person who uses mime as a theatrical medium or as a performance art Performance art is an artwork or art exhibition created through actions executed ...

MIME
) parlance is the same as a character encoding (but not the same as CCS). "
Legacy In law, a legacy is something held and transferred to someone as their inheritance Inheritance is the practice of passing on private property, Title (property), titles, debts, entitlements, Privilege (law), privileges, rights, and Law of obligat ...
encoding" is a term sometimes used to characterize old character encodings, but with an ambiguity of sense. Most of its use is in the context of , where it refers to encodings that fail to cover all Unicode code points, or, more generally, using a somewhat different character repertoire: several code points representing one Unicode character, or versa (see e.g.
code page 437 Code page 437 (CCSID A CCSID (coded character set identifier) is a 16-bit number that represents a particular encoding In communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, mea ...
). Some sources refer to an encoding as ''legacy'' only because it preceded Unicode. All Windows code pages are usually referred to as legacy, both because they antedate Unicode and because they are unable to represent all 221 possible Unicode code points.


Character encoding translation

As a result of having many character encoding methods in use (and the need for backward compatibility with archived data), many computer programs have been developed to translate data between encoding schemes as a form of data
transcoding Transcoding is the direct digital-to-digital conversion of one encoding In communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, meaning among Subject (philosophy), entities or Org ...
. Some of these are cited below.
Cross-platform In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes and development of both computer hardware , hardware and soft ...

Cross-platform
: *
Web browser A web browser (commonly referred to as a browser) is application software for accessing the World Wide Web. When a User (computing), user requests a web page from a particular website, the web browser retrieves the necessary content from a web ...

Web browser
s – most modern web browsers feature automatic character encoding detection. On Firefox 3, for example, see the View/Character Encoding submenu. * iconv – program and standardized API to convert encodings * luit – program that converts encoding of input and output to programs running interactively * convert_encoding.py – Python based utility to convert text files between arbitrary encodings and line endings. * decodeh.py – algorithm and module to heuristically guess the encoding of a string. *
International Components for Unicode International Components for Unicode (ICU) is an open-source software, open-source project of mature C (programming language), C/C++ and Java (programming language), Java libraries for Unicode support, software Internationalization and localizati ...
– A set of C and Java libraries to perform charset conversion. uconv can be used from ICU4C.
chardet
– This is a translation of the
Mozilla Mozilla (stylized as moz://a) is a free software Free software (or libre software) is computer software distributed under terms that allow users to run the software for any purpose as well as to study, change, and distribute it and any a ...

Mozilla
automatic-encoding-detection code into the Python computer language. * The newer versions of the Unix
file File or filing may refer to: Mechanical tools and processes * File (tool) A file is a tool used to remove fine amounts of material from a workpiece. It is common in woodworking, metalworking, and other similar trade and hobby tasks. Most are ...
command attempt to do a basic detection of character encoding (also available on
Cygwin Cygwin ( ) is a POSIX The Portable Operating System Interface (POSIX) is a family of standard Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, conventio ...
).
charset
C++ C++ () is a general-purpose programming language In computer software, a general-purpose programming language is a programming language dedicated to a general-purpose, designed to be used for writing software in a wide variety of application ...

C++
template library with simple interface to convert between C++/user-defined streams. charset defined many character-sets and allows you to use Unicode formats with support of
endianness In computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithm of an algorithm (Euclid's algorithm) for calculating the greatest comm ...
.
Unix-like A Unix-like (sometimes referred to as UN*X or *nix) operating system An operating system (OS) is system software System software is software designed to provide a platform for other software. Examples of system software include operating s ...
: * cmv – simple tool for transcoding filenames. * convmv – convert a filename from one encoding to another. * cstocs – convert file contents from one encoding to another for the Czech and Slovak languages. * enca – analyzes encodings for given text files. * recode – convert file contents from one encoding to another * utrac – convert file contents from one encoding to another.
Windows Microsoft Windows, commonly referred to as Windows, is a group of several proprietary {{Short pages monitor