![Papertape-Wikipedia-example-dark1-2000px](https://upload.wikimedia.org/wikipedia/commons/6/68/Papertape-Wikipedia-example-dark1-2000px.png)
Character encoding is the process of assigning numbers to
graphical
Graphics () are visual images or designs on some surface, such as a wall, canvas, screen, paper, or stone, to inform, illustrate, or entertain. In contemporary usage, it includes a pictorial representation of data, as in design and manufacture ...
characters
Character or Characters may refer to:
Arts, entertainment, and media Literature
* ''Character'' (novel), a 1936 Dutch novel by Ferdinand Bordewijk
* ''Characters'' (Theophrastus), a classical Greek set of character sketches attributed to The ...
, especially the written characters of
human language
Language is a structured system of communication. The structure of a language is its grammar and the free components are its vocabulary. Languages are the primary means by which humans communicate, and may be conveyed through a variety of met ...
, allowing them to be
stored,
transmitted, and
transformed using
digital computers. The numerical values that make up a character encoding are known as "
code point
In character encoding terminology, a code point, codepoint or code position is a numerical value that maps to a specific character. Code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but ...
s" and collectively comprise a "code space", a "
code page", or a "
character map
Character Map is a utility included with Microsoft Windows operating systems and is used to view the characters in any installed font, to check what keyboard input ( Alt code) is used to enter those characters, and to copy characters to the cli ...
".
Early character codes associated with the optical or electrical
telegraph
Telegraphy is the long-distance transmission of messages where the sender uses symbolic codes, known to the recipient, rather than a physical exchange of an object bearing the message. Thus flag semaphore is a method of telegraphy, whereas p ...
could only represent a subset of the characters used in
written language
A written language is the representation of a spoken or gestural language by means of a writing system. Written language is an invention in that it must be taught to children, who will pick up spoken language or sign language by exposure eve ...
s, sometimes restricted to
upper case letters,
numerals and some
punctuation
Punctuation (or sometimes interpunction) is the use of spacing, conventional signs (called punctuation marks), and certain typographical devices as aids to the understanding and correct reading of written text, whether read silently or aloud. An ...
only. The low cost of digital representation of data in modern computer systems allows more elaborate character codes (such as
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
) which represent most of the characters used in many written languages. Character encoding using internationally accepted standards permits worldwide interchange of text in electronic form.
History
The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance, using once-novel electrical means. The earliest codes were based upon manual and hand-written encoding and cyphering systems, such as
Bacon's cipher
Bacon's cipher or the Baconian cipher is a method of steganographic message encoding devised by Francis Bacon in 1605. A message is concealed in the presentation of text, rather than its content.
Cipher details
To encode a message, each letter of ...
,
Braille
Braille (Pronounced: ) is a tactile writing system used by people who are visually impaired, including people who are blind, deafblind or who have low vision. It can be read either on embossed paper or by using refreshable braille disp ...
,
International maritime signal flags, and the 4-digit encoding of Chinese characters for a
Chinese telegraph code
The Chinese telegraph code, Chinese telegraphic code, or Chinese commercial code ( or ) is a four-digit decimal code (character encoding) for electrically telegraphing messages written with Chinese characters.
Encoding and decoding
A codebook ...
(
Hans Schjellerup
Hans Carl Frederik Christian Schjellerup (8 February 1827 – 13 November 1887) was a Danish astronomer.
He was born at Odense, the son of a jeweller. Initially he was apprenticed as a watch maker, but in 1848 he passed the entrance exam for th ...
, 1869). With the adoption of electrical and electro-mechanical techniques these earliest codes were adapted to the new capabilities and limitations of the early machines. The earliest well-known electrically transmitted character code,
Morse code, introduced in the 1840s, used a system of four "symbols" (short signal, long signal, short space, long space) to generate codes of variable length. Though some commercial use of Morse code was via machinery, it was often used as a manual code, generated by hand on a
telegraph key
A telegraph key is a specialized electrical switch used by a trained operator to transmit text messages in Morse code in a telegraphy system. Keys are used in all forms of electrical telegraph systems, including landline (also called wire) ...
and decipherable by ear, and persists in
amateur radio and
aeronautical use. Most codes are of fixed per-character length or variable-length sequences of fixed-length codes (e.g.
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
).
Common examples of character encoding systems include
Morse code, the
Baudot code
The Baudot code is an early character encoding for telegraphy invented by Émile Baudot in the 1870s. It was the predecessor to the International Telegraph Alphabet No. 2 (ITA2), the most common teleprinter code in use until the advent of ASCII ...
, the American Standard Code for Information Interchange (
ASCII
ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because ...
) and
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
.
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
, a well defined and extensible encoding system, has supplanted most earlier character encodings, but the path of code development to the present is fairly well known.
The
Baudot code
The Baudot code is an early character encoding for telegraphy invented by Émile Baudot in the 1870s. It was the predecessor to the International Telegraph Alphabet No. 2 (ITA2), the most common teleprinter code in use until the advent of ASCII ...
, a five-bit encoding, was created by
Émile Baudot
Jean-Maurice-Émile Baudot (; 11 September 1845 – 28 March 1903), French telegraph engineer and inventor of the first means of digital communication Baudot code, was one of the pioneers of telecommunications. He invented a multiplexed printi ...
in 1870, patented in 1874, modified by Donald Murray in 1901, and standardized by CCITT as International Telegraph Alphabet No. 2 (ITA2) in 1930. The name "baudot" has been erroneously applied to ITA2 and its many variants. ITA2 suffered from many shortcomings and was often "improved" by many equipment manufacturers, sometimes creating compatibility issues. In 1959 the U.S. military defined its
Fieldata code, a six-or seven-bit code, introduced by the U.S. Army Signal Corps. While Fieldata addressed many of the then-modern issues (e.g. letter and digit codes arranged for machine collation), Fieldata fell short of its goals and was short-lived. In 1963 the first ASCII (American Standard Code for Information Interchange) code was released (X3.4-1963) by the ASCII committee (which contained at least one member of the Fieldata committee, W. F. Leubbert) which addressed most of the shortcomings of Fieldata, using a simpler code. Many of the changes were subtle, such as collatable character sets within certain numeric ranges. ASCII63 was a success, widely adopted by industry, and with the follow-up issue of the 1967 ASCII code (which added lower-case letters and fixed some "control code" issues) ASCII67 was adopted fairly widely. ASCII67's American-centric nature was somewhat addressed in the European
ECMA-6
ISO/IEC 646 is a set of ISO/IEC standards, described as ''Information technology — ISO 7-bit coded character set for information interchange'' and developed in cooperation with ASCII at least since 1964. Since its first edition in ...
standard.
Herman Hollerith
Herman Hollerith (February 29, 1860 – November 17, 1929) was a German-American statistician, inventor, and businessman who developed an electromechanical tabulating machine for punched cards to assist in summarizing information and, later, i ...
invented punch card data encoding in the late 19th century to analyze census data. Initially, each hole position represented a different data element, but later, numeric information was encoded by numbering the lower rows 0 to 9, with a punch in a column representing its row number. Later alphabetic data was encoded by allowing more than one punch per column. Electromechanical
tabulating machine
The tabulating machine was an electromechanical machine designed to assist in summarizing information stored on punched cards. Invented by Herman Hollerith, the machine was developed to help process data for the 1890 U.S. Census. Later model ...
s represented date internally by the timing of pulses relative to the motion of the cards through the machine. When
IBM went to electronic processing, starting with the
IBM 603 Electronic Multiplier, it used a variety of binary encoding schemes that were tied to the punch card code.
IBM's
Binary Coded Decimal
In computing and electronic systems, binary-coded decimal (BCD) is a class of binary encodings of decimal numbers where each digit is represented by a fixed number of bits, usually four or eight. Sometimes, special bit patterns are used fo ...
(
BCD) was a six-bit encoding scheme used by IBM as early as 1953 in its
702
__NOTOC__
Year 702 ( DCCII) was a common year starting on Sunday (link will display the full calendar) of the Julian calendar. The denomination 702 for this year has been used since the early medieval period, when the Anno Domini calendar era b ...
and
704
__NOTOC__
Year 704 ( DCCIV) was a leap year starting on Tuesday (link will display the full calendar) of the Julian calendar. The denomination 704 for this year has been used since the early medieval period, when the Anno Domini calendar era be ...
computers, and in its later
7000 Series 7000 series may refer to:
Japanese trains
* Chichibu Railway 7000 series electric multiple unit (EMU)
* Echizen Railway 7000 series EMU
* Hankyu 7000 series EMU
* Hokushin Kyuko Electric Railway 7000 series EMU operating for the Kobe Municipal Su ...
and
1400 series, as well as in associated peripherals. Since the punched card code then in use only allowed digits, upper-case English letters and a few special characters, six bits were sufficient. BCD extended existing simple four-bit numeric encoding to include alphabetic and special characters, mapping it easily to punch-card encoding which was already in widespread use. IBMs codes were used primarily with IBM equipment; other computer vendors of the era had their own character codes, often six-bit, but usually had the ability to read tapes produced on IBM equipment. BCD was the precursor of
IBM's
Extended Binary Coded Decimal Interchange Code
Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding six- ...
(usually abbreviated as EBCDIC), an eight-bit encoding scheme developed in 1963 for the
IBM System/360 that featured a larger character set, including lower case letters.
The limitations of such sets soon became apparent, and a number of ''ad hoc'' methods were developed to extend them. The need to support more
writing system
A writing system is a method of visually representing verbal communication, based on a script and a set of rules regulating its use. While both writing and speech are useful in conveying messages, writing differs in also being a reliable fo ...
s for different languages, including the
CJK family of
East Asian scripts, required support for a far larger number of characters and demanded a systematic approach to character encoding rather than the previous ''ad hoc'' approaches.
In trying to develop universally interchangeable character encodings, researchers in the 1980s faced the dilemma that, on the one hand, it seemed necessary to add more bits to accommodate additional characters, but on the other hand, for the users of the relatively small character set of the Latin alphabet (who still constituted the majority of computer users), those additional bits were a colossal waste of then-scarce and expensive computing resources (as they would always be zeroed out for such users). In 1985, the average personal computer user's
hard disk drive
A hard disk drive (HDD), hard disk, hard drive, or fixed disk is an electro-mechanical data storage device that stores and retrieves digital data using magnetic storage with one or more rigid rapidly rotating platters coated with magne ...
could store only about 10 megabytes, and it cost approximately US$250 on the wholesale market (and much higher if purchased separately at retail),
so it was very important at the time to make every bit count.
The compromise solution that was eventually found and developed into Unicode was to break the assumption (dating back to telegraph codes) that each character should always directly correspond to a particular sequence of bits. Instead, characters would first be mapped to a universal intermediate representation in the form of abstract numbers called
code point
In character encoding terminology, a code point, codepoint or code position is a numerical value that maps to a specific character. Code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but ...
s. Code points would then be represented in a variety of ways and with various default numbers of bits per character (code units) depending on context. To encode code points higher than the length of the code unit, such as above 256 for eight-bit units, the solution was to implement
variable-length encodings where an escape sequence would signal that subsequent bits should be parsed as a higher code point.
Terminology
; Terminology related to character encoding:
![KB Dubeolsik for Old Hangul (NG3)](https://upload.wikimedia.org/wikipedia/commons/c/c9/KB_Dubeolsik_for_Old_Hangul_%28NG3%29.svg)
* A ''character'' is a minimal unit of text that has semantic value.
* A ''character set'' is a collection of characters that might be used by multiple languages. ''Example:'' The Latin character set is used by English and most European languages, though the Greek character set is used only by the Greek language.
* A ''coded character set'' is a character set in which each character corresponds to a unique number.
* A ''code point'' of a coded character set is any allowed value in the character set or code space.
* A ''code space'' is a range of integers whose values are code points.
* A ''code unit'' is the "word size" of the character encoding scheme, such as 7-bit, 8-bit, 16-bit. In some schemes, some characters are encoded using multiple code units, resulting in a variable-length encoding. A code unit is referred to as a ''code value'' in some documents.
; Character repertoire (the abstract set of characters):
The character repertoire is an abstract set of more than one million characters found in a wide variety of scripts including Latin, Cyrillic, Chinese, Korean, Japanese, Hebrew, and Aramaic.
Other symbols such as musical notation are also included in the character repertoire. Both the Unicode and
GB 18030
GB 18030 is a Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China. GB18030 is the registered Internet n ...
standards have a character repertoire. As new characters are added to one standard, the other standard also adds those characters, to maintain parity.
The code unit size is equivalent to the bit measurement for the particular encoding:
* A code unit in
US-ASCII
ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because of ...
consists of 7 bits;
* A code unit in
UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of ...
,
EBCDIC
Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight- bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding ...
and GB 18030 consists of 8 bits;
* A code unit in
UTF-16
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as cod ...
consists of 16 bits;
* A code unit in
UTF-32
UTF-32 (32- bit Unicode Transformation Format) is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode ...
consists of 32 bits.
; Example of a code unit:
Consider a
string of the letters "ab̲c𐐀", that is, a string containing a Unicode combining character () as well a supplementary character (). This string has several representions which are logically equivalent, yet while each is suited to a diverse set of circumstances or range of requirements:
* Four
composed characters:
*:, , ,
* Five
grapheme
In linguistics, a grapheme is the smallest functional unit of a writing system.
The word ''grapheme'' is derived and the suffix ''-eme'' by analogy with ''phoneme'' and other names of emic units. The study of graphemes is called '' graphemi ...
s:
*:, , , ,
* Five Unicode
code point
In character encoding terminology, a code point, codepoint or code position is a numerical value that maps to a specific character. Code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but ...
s:
*:, , , ,
* Five UTF-32 code units (32-bit integer values):
*:, , , ,
* Six UTF-16 code units (16-bit integers)
*:, , , , ,
* Nine UTF-8 code units (8-bit values, or
byte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
s)
*:, , , , , , , ,
Note in particular the last character, which is represented with either one ''1'' 32-bit value, ''2'' 16-bit values. or ''4'' 8-bit values. Although each of those forms uses the same total number of bits (32) to represent the glyph, the actual numeric byte values and their arrangement appear entirely unrelated.
; Code point:
The convention to refer to a character in Unicode is to start with 'U+' followed by the codepoint value in hexadecimal. The range of valid code points for the Unicode standard is U+0000 to U+10FFFF, inclusive, divided in 17
planes, identified by the numbers 0 to 16. Characters in the range U+0000 to U+FFFF are in plane 0, called the
Basic Multilingual Plane
In the Unicode standard, a plane is a continuous group of 65,536 (216) code points. There are 17 planes, identified by the numbers 0 to 16, which corresponds with the possible values 00–1016 of the first two positions in six position hexadecima ...
(BMP). This plane contains most commonly-used characters. Characters in the range U+10000 to U+10FFFF in the other planes are called
supplementary characters.
The following table shows examples of code point values:
A code point is represented by a sequence of code units. The mapping is defined by the encoding. Thus, the number of code units required to represent a code point depends on the encoding:
* ''UTF-8:'' code points map to a sequence of one, two, three or four code units.
* ''UTF-16:'' code units are twice as long as 8-bit code units. Therefore, any code point with a scalar value less than U+10000 is encoded with a single code unit. Code points with a value U+10000 or higher require two code units each. These pairs of code units have a unique term in UTF-16:
"Unicode surrogate pairs".
* ''UTF-32:'' the 32-bit code unit is large enough that every code point is represented as a single code unit.
* ''GB 18030:'' multiple code units per code point are common, because of the small code units. Code points are mapped to one, two, or four code units.
Unicode encoding model
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
and its parallel standard, the ISO/IEC 10646
Universal Character Set
The Universal Coded Character Set (UCS, Unicode) is a standard set of characters defined by the international standard ISO/IEC 10646, ''Information technology — Universal Coded Character Set (UCS)'' (plus amendments to that standard), w ...
, together constitute a modern, unified character encoding. Rather than mapping characters directly to octets (
byte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable uni ...
s), they separately define what characters are available, corresponding natural numbers (
code point
In character encoding terminology, a code point, codepoint or code position is a numerical value that maps to a specific character. Code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but ...
s), how those numbers are encoded as a series of fixed-size natural numbers (code units), and finally how those units are encoded as a stream of octets. The purpose of this decomposition is to establish a universal set of characters that can be encoded in a variety of ways.
To describe this model correctly requires more precise terms than "character set" and "character encoding." The terms used in the modern model follow:
[
A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, i.e. no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the ]Windows code page
Windows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode was implemented in Wind ...
s). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into basic information units. The basic variants of the Latin
Latin (, or , ) is a classical language belonging to the Italic branch of the Indo-European languages. Latin was originally a dialect spoken in the lower Tiber area (then known as Latium) around present-day Rome, but through the power of the ...
, Greek
Greek may refer to:
Greece
Anything of, from, or related to Greece, a country in Southern Europe:
*Greeks, an ethnic group.
*Greek language, a branch of the Indo-European language family.
**Proto-Greek language, the assumed last common ancestor ...
and Cyrillic alphabets can be broken down into letters, digits, punctuation, and a few ''special characters'' such as the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. But even with these alphabets, diacritic
A diacritic (also diacritical mark, diacritical point, diacritical sign, or accent) is a glyph added to a letter or to a basic glyph. The term derives from the Ancient Greek (, "distinguishing"), from (, "to distinguish"). The word ''diacriti ...
s pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Ligatures pose similar problems. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined in different ways for different situations.
A coded character set (CCS) is a function
Function or functionality may refer to:
Computing
* Function key, a type of key on computer keyboards
* Function model, a structured representation of processes in a system
* Function object or functor or functionoid, a concept of object-oriente ...
that maps characters to ''code point
In character encoding terminology, a code point, codepoint or code position is a numerical value that maps to a specific character. Code points usually represent a single grapheme—usually a letter, digit, punctuation mark, or whitespace—but ...
s'' (each code point represents one character). For example, in a given repertoire, the capital letter "A" in the Latin alphabet might be represented by the code point 65, the character "B" to 66, and so on. Multiple coded character sets may share the same repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different code points.
A character encoding form (CEF) is the mapping of code points to ''code units'' to facilitate storage in a system that represents numbers as bit sequences of fixed length (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units can only directly represent code points 0 to 65,535 in each unit, but larger code points (say, 65,536 to 1.4 million) could be represented by using multiple 16-bit units. This correspondence is defined by a CEF.
Next, a character encoding scheme (CES) is the mapping of code units to a sequence of octets to facilitate storage on an octet-based file system or transmission over an octet-based network. Simple character encoding schemes include UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of ...
, UTF-16BE, UTF-32BE, UTF-16LE or UTF-32LE; compound character encoding schemes, such as UTF-16
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as cod ...
, UTF-32
UTF-32 (32- bit Unicode Transformation Format) is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode ...
and ISO/IEC 2022
ISO/IEC 2022 ''Information technology—Character code structure and extension techniques'', is an ISO/IEC standard (equivalent to the ECMA standard ECMA-35, the ANSI standard ANSI X3.41 and the Japanese Industrial Standard JIS X 0202) in the ...
, switch between several simple schemes by using a byte order mark
The byte order mark (BOM) is a particular usage of the special Unicode character, , whose appearance as a magic number at the start of a text stream can signal several things to a program reading the text:
* The byte order, or endianness, of t ...
or escape sequence
In computer science, an escape sequence is a combination of characters that has a meaning other than the literal characters contained therein; it is marked by one or more preceding (and possibly terminating) characters.
Examples
* In C and ma ...
s; compressing schemes try to minimize the number of bytes used per code unit (such as SCSU, BOCU, and Punycode
Punycode is a representation of Unicode with the limited ASCII character subset used for Internet hostnames. Using Punycode, host names containing Unicode characters are transcoded to a subset of ASCII consisting of letters, digits, and hyphens, wh ...
).
Although UTF-32BE is a simpler CES, most systems working with Unicode use either UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of ...
, which is backward compatible
Backward compatibility (sometimes known as backwards compatibility) is a property of an operating system, product, or technology that allows for interoperability with an older legacy system, or with input designed for such a system, especially in ...
with fixed-length ASCII and maps Unicode code points to variable-length sequences of octets, or UTF-16BE, which is backward compatible
Backward compatibility (sometimes known as backwards compatibility) is a property of an operating system, product, or technology that allows for interoperability with an older legacy system, or with input designed for such a system, especially in ...
with fixed-length UCS-2BE and maps Unicode code points to variable-length sequences of 16-bit words. See comparison of Unicode encodings for a detailed discussion.
Finally, there may be a higher-level protocol which supplies additional information to select the particular variant of a Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
character, particularly where there are regional variants that have been 'unified' in Unicode as the same character. An example is the XML
Extensible Markup Language (XML) is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable ...
attribute xml:lang.
The Unicode model uses the term character map for historical systems which directly assign a sequence of characters to a sequence of bytes, covering all of CCS, CEF and CES layers.
Character sets, character maps and code pages
Historically, the terms "character encoding", "character map", "character set" and " code page" were synonymous in computer science
Computer science is the study of computation, automation, and information. Computer science spans theoretical disciplines (such as algorithms, theory of computation, information theory, and automation) to practical disciplines (includi ...
, as the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units – usually with a single character per code unit. But now the terms have related but distinct meanings, due to efforts by standards bodies to use precise terminology when writing about and unifying many different encoding systems.[ Regardless, the terms are still used interchangeably, with ''character set'' being nearly ubiquitous.
A "code page" usually means a ]byte-oriented
Byte-oriented framing protocol is "a communications protocol in which full bytes are used as control codes. Also known as character-oriented protocol." For example UART communication is byte-oriented.
The term "character-oriented" is deprecated, ...
encoding, but with regard to some suite of encodings (covering different scripts), where many characters share the same codes
In communications and information processing, code is a system of rules to convert information—such as a letter, word, sound, image, or gesture—into another form, sometimes shortened or secret, for communication through a communication ...
in most or all those code pages. Well-known code page suites are "Windows" (based on Windows-1252) and "IBM"/"DOS" (based on code page 437), see Windows code page
Windows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode was implemented in Wind ...
for details. Most, but not all, encodings referred to as code pages are single-byte encodings (but see octet
Octet may refer to:
Music
* Octet (music), ensemble consisting of eight instruments or voices, or composition written for such an ensemble
** String octet, a piece of music written for eight string instruments
*** Octet (Mendelssohn), 1825 compos ...
on byte size.)
IBM's Character Data Representation Architecture (CDRA) designates entities with coded character set identifiers (CCSID
A CCSID (coded character set identifier) is a 16-bit number that represents a particular encoding of a specific code page. For example, Unicode is a code page that has several encoding (so called "transformation") forms, like UTF-8, UTF-16 and U ...
s), each of which is variously called a "charset", "character set", "code page", or "CHARMAP".[
The term "code page" does not occur in Unix or Linux where "charmap" is preferred, usually in the larger context of locales.
In contrast to a " coded character set", a "character encoding" is a map from abstract characters to ]code word
In communication, a code word is an element of a standardized code or protocol. Each code word is assembled in accordance with the specific rules of the code and assigned a unique meaning. Code words are typically used for reasons of reliability, ...
s. A "character set" in HTTP
The Hypertext Transfer Protocol (HTTP) is an application layer protocol in the Internet protocol suite model for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide We ...
(and MIME
Multipurpose Internet Mail Extensions (MIME) is an Internet standard that extends the format of email messages to support text in character sets other than ASCII, as well as attachments of audio, video, images, and application programs. Message ...
) parlance is the same as a character encoding (but not the same as CCS).
" Legacy encoding" is a term sometimes used to characterize old character encodings, but with an ambiguity of sense. Most of its use is in the context of Unicodification, where it refers to encodings that fail to cover all Unicode code points, or, more generally, using a somewhat different character repertoire: several code points representing one Unicode character, or versa (see e.g. code page 437). Some sources refer to an encoding as ''legacy'' only because it preceded Unicode. All Windows code pages are usually referred to as legacy, both because they antedate Unicode and because they are unable to represent all 221 possible Unicode code points.
Character encoding translation
As a result of having many character encoding methods in use (and the need for backward compatibility with archived data), many computer programs have been developed to translate data between encoding schemes as a form of data transcoding
Transcoding is the direct digital-to-digital conversion of one encoding to another, such as for video data files, audio files (e.g., MP3, WAV), or character encoding (e.g., UTF-8, ISO/IEC 8859). This is usually done in cases where a target d ...
. Some of these are cited below.
Cross-platform
In computing, cross-platform software (also called multi-platform software, platform-agnostic software, or platform-independent software) is computer software that is designed to work in several computing platforms. Some cross-platform software ...
:
* Web browser
A web browser is application software for accessing websites. When a user requests a web page from a particular website, the browser retrieves its files from a web server and then displays the page on the user's screen. Browsers are used o ...
s – most modern web browsers feature automatic character encoding detection. On Firefox 3, for example, see the View/Character Encoding submenu.
* iconv
In Unix and Unix-like operating systems, iconv (an abbreviation of internationalization conversion) is a command-line program and a standardized application programming interface (API) used to convert between different character encodings. "It ...
– a program and standardized API to convert encodings
* luit – a program that converts encoding of input and output to programs running interactively
* convert_encoding.py – a Python-based utility to convert text files between arbitrary encodings and line endings
* decodeh.py – an algorithm and module to heuristically guess the encoding of a string
* International Components for Unicode
International Components for Unicode (ICU) is an open-source project of mature C/ C++ and Java libraries for Unicode support, software internationalization, and software globalization. ICU is widely portable to many operating systems and environ ...
– A set of C and Java libraries to perform charset conversion. uconv can be used from ICU4C.
chardet
– This is a translation of the Mozilla
Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape. The Mozilla community uses, develops, spreads and supports Mozilla products, thereby promoting exclusively free software and open standards, w ...
automatic-encoding-detection code into the Python computer language.
* The newer versions of the Unix file
File or filing may refer to:
Mechanical tools and processes
* File (tool), a tool used to ''remove'' fine amounts of material from a workpiece
**Filing (metalworking), a material removal process in manufacturing
** Nail file, a tool used to gent ...
command attempt to do a basic detection of character encoding (also available on Cygwin).
charset
– C++
C++ (pronounced "C plus plus") is a high-level general-purpose programming language created by Danish computer scientist Bjarne Stroustrup as an extension of the C programming language, or "C with Classes". The language has expanded significan ...
template library with simple interface to convert between C++/user-defined streams. charset defined many character-sets and allows you to use Unicode formats with support of endianness
In computing, endianness, also known as byte sex, is the order or sequence of bytes of a word of digital data in computer memory. Endianness is primarily expressed as big-endian (BE) or little-endian (LE). A big-endian system stores the mos ...
.
Unix-like
A Unix-like (sometimes referred to as UN*X or *nix) operating system is one that behaves in a manner similar to a Unix system, although not necessarily conforming to or being certified to any version of the Single UNIX Specification. A Unix-li ...
:
* cmv – a simple tool for transcoding filenames.
* convmv – converts a filename from one encoding to another.
* cstocs – converts file contents from one encoding to another for the Czech and Slovak languages.
* enca – analyzes encodings for given text files.
* recode – converts file contents from one encoding to another.
* utrac – converts file contents from one encoding to another.
Windows
Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft. Each family caters to a certain sector of the computing industry. For example, Windows NT for consumers, Windows Server for ser ...
:
* Encoding.Convert – .NET API
* MultiByteToWideChar/WideCharToMultiByte – to convert from ANSI to Unicode & Unicode to ANSI
* cscvt – a character set conversion tool
* enca – analyzes encodings for given text files.
See also
* Percent-encoding
Percent-encoding, also known as URL encoding, is a method to encode arbitrary data in a Uniform Resource Identifier (URI) using only the limited US-ASCII characters legal within a URI. Although it is known as ''URL encoding'', it is also used ...
* Alt code
* Character encodings in HTML
While Hypertext Markup Language (HTML) has been in use since 1991, HTML 4.0 from December 1997 was the first standardized version where international characters were given reasonably complete treatment. When an HTML document includes special ch ...
* :Character encoding – articles related to character encoding in general
* :Character sets – articles detailing specific character encodings
* Hexadecimal representations
* ''Mojibake
Mojibake ( ja, 文字化け; , "character transformation") is the garbled text that is the result of text being decoded using an unintended character encoding. The result is a systematic replacement of symbols with completely unrelated ones, oft ...
'' – character set mismap
* Mojikyō – a system ("glyph set") that includes over 100,000 Chinese character drawings, modern and ancient, popular and obscure
* Presentation layer
In the seven-layer OSI model of computer networking, the presentation layer is layer 6 and serves as the data translator for the network. It is sometimes called the syntax layer.
Description
Within the service layering semantics of the OSI netw ...
* TRON
''Tron'' (stylized as ''TRON'') is a 1982 American science fiction action- adventure film written and directed by Steven Lisberger from a story by Lisberger and Bonnie MacBird. The film stars Jeff Bridges as Kevin Flynn, a computer programmer ...
, part of the TRON project, is an encoding system that does not use Han Unification; instead, it uses "control codes" to switch between 16-bit "planes" of characters.
* Universal Character Set characters
The Unicode Consortium and the ISO/IEC JTC 1/SC 2/ WG 2 jointly collaborate on the list of the characters in the Universal Coded Character Set. The Universal Coded Character Set, most commonly called the Universal Character Set ( UCS, officia ...
* Charset sniffing – used in some applications when character encoding metadata is not available
Common character encodings
* ISO 646
ISO/IEC 646 is a set of ISO/IEC standards, described as ''Information technology — ISO 7-bit coded character set for information interchange'' and developed in cooperation with ASCII at least since 1964. Since its first edition in ...
** ASCII
ASCII ( ), abbreviated from American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because ...
* EBCDIC
Extended Binary Coded Decimal Interchange Code (EBCDIC; ) is an eight- bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding ...
* ISO 8859
ISO/IEC 8859 is a joint ISO and IEC series of standards for 8-bit character encodings. The series of standards consists of numbered parts, such as ISO/IEC 8859-1, ISO/IEC 8859-2, etc. There are 15 parts, excluding the abandoned ISO/IEC 8859-12. ...
:
** ISO 8859-1 Western Europe
** ISO 8859-2
ISO/IEC 8859-2:1999, ''Information technology — 8-bit single-byte coded graphic character sets — Part 2: Latin alphabet No. 2'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1987. ...
Western and Central Europe
** ISO 8859-3
ISO/IEC 8859-3:1999, ''Information technology — 8-bit single-byte coded graphic character sets — Part 3: Latin alphabet No. 3'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1988. I ...
Western Europe and South European (Turkish, Maltese plus Esperanto)
** ISO 8859-4
ISO/IEC 8859-4:1998, ''Information technology — 8-bit single-byte coded graphic character sets — Part 4: Latin alphabet No. 4'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1988. I ...
Western Europe and Baltic countries (Lithuania, Estonia, Latvia and Lapp)
** ISO 8859-5
ISO/IEC 8859-5:1999, ''Information technology — 8-bit single-byte coded graphic character sets — Part 5: Latin/Cyrillic alphabet'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 198 ...
Cyrillic alphabet
** ISO 8859-6 Arabic
** ISO 8859-7
ISO is the most common abbreviation for the International Organization for Standardization.
ISO or Iso may also refer to: Business and finance
* Iso (supermarket), a chain of Danish supermarkets incorporated into the SuperBest chain in 2007
* Iso ...
Greek
** ISO 8859-8
ISO/IEC 8859-8, ''Information technology — 8-bit single-byte coded graphic character sets — Part 8: Latin/Hebrew alphabet'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings. ISO/IEC 8859-8:1999 from 1999 represen ...
Hebrew
** ISO 8859-9
ISO/IEC 8859-9:1999, ''Information technology — 8-bit single-byte coded graphic character sets — Part 9: Latin alphabet No. 5'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1989. ...
Western Europe with amended Turkish character set
** ISO 8859-10
ISO/IEC 8859-10:1998, ''Information technology — 8-bit single-byte coded graphic character sets — Part 10: Latin alphabet No. 6'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1992. ...
Western Europe with rationalised character set for Nordic languages, including complete Icelandic set
** ISO 8859-11
ISO/IEC 8859-11:2001, ''Information technology — 8-bit single-byte coded graphic character sets — Part 11: Latin/Thai alphabet'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 2001. I ...
Thai
** ISO 8859-13
ISO/IEC 8859-13:1998, ''Information technology — 8-bit single-byte coded graphic character sets — Part 13: Latin alphabet No. 7'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1998. ...
Baltic languages plus Polish
** ISO 8859-14
ISO/IEC 8859-14:1998, ''Information technology — 8-bit single-byte coded graphic character sets — Part 14: Latin alphabet No. 8 (Celtic)'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published ...
Celtic languages (Irish Gaelic, Scottish, Welsh)
** ISO 8859-15
ISO/IEC 8859-15:1999, ''Information technology — 8-bit single-byte coded graphic character sets — Part 15: Latin alphabet No. 9'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1999. ...
Added the Euro sign and other rationalisations to ISO 8859-1
** ISO 8859-16
ISO/IEC 8859-16:2001, ''Information technology — 8-bit single-byte coded graphic character sets — Part 16: Latin alphabet No. 10'', is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 2001 ...
Central, Eastern and Southern European languages (Albanian, Bosnian, Croatian, Hungarian, Polish, Romanian, Serbian and Slovenian, but also French, German, Italian and Irish Gaelic)
* CP437, CP720, CP737, CP850
Code page 850 (CCSID 850) (also known as CP 850, IBM 00850, OEM 850, DOS Latin 1) is a code page used under DOS and Psion's EPOC16 operating systems in Western Europe. Depending on the country setting and system configuration, code page 850 is ...
, CP852, CP855, CP857, CP858, CP860, CP861, CP862, CP863, CP865, CP866, CP869, CP872
* MS-Windows character sets:
** Windows-1250
Windows-1250 is a code page used under Microsoft Windows to represent texts in Central European and Eastern European languages that use Latin script, such as Czech (which is its main user with half its use, though Czech has 96.6% use of UTF-8, an ...
for Central European languages that use Latin script, (Polish, Czech, Slovak, Hungarian, Slovene, Serbian, Croatian, Bosnian, Romanian and Albanian)
** Windows-1251
Windows-1251 is an 8-bit character encoding, designed to cover languages that use the Cyrillic script such as Russian, Ukrainian, Belarusian, Bulgarian, Serbian Cyrillic, Macedonian and other languages.
On the web, it is the second most-used ...
for Cyrillic alphabets
** Windows-1252
Windows-1252 or CP-1252 ( code page 1252) is a single-byte character encoding of the Latin alphabet, used by default in the legacy components of Microsoft Windows for English and many European languages including Spanish, French, and German.
I ...
for Western languages
** Windows-1253
Windows code page 1253 ("Greek - ANSI"), commonly known by its IANA-registered name Windows-1253 or abbreviated as cp1253, is a Microsoft Windows code page used to write modern Greek. It is not capable of supporting the older polytonic Greek.
It i ...
for Greek
** Windows-1254
Windows-1254 is a code page used under Microsoft Windows (and for the web), to write Turkish that it was designed for (which is its dominant user, even though it can be used for some other languages too). Characters with codepoints A0 through F ...
for Turkish
** Windows-1255
Windows-1255 is a code page used under Microsoft Windows to write Hebrew. It is an almost compatible superset of ISO-8859-8 most of the symbols are in the same positions (except for A4, which is 'sheqel sign' in Windows-1255 but 'generic currency ...
for Hebrew
** Windows-1256 for Arabic
** Windows-1257
Windows-1257 (Windows Baltic) is an 8-bit, single-byte extended ASCII code page used to support the Estonian, Latvian and Lithuanian languages under Microsoft Windows. In Lithuania, it is standardised as LST 1590-3, alongside a modified variant ...
for Baltic languages
** Windows-1258
Windows-1258 is a code page used in Microsoft Windows to represent Vietnamese texts. It makes use of combining diacritical marks.
Windows-1258 is compatible with neither the Vietnamese standard ( TCVN 5712 / VSCII), nor the various other encodin ...
for Vietnamese
* Mac OS Roman
Mac OS Roman is a character encoding created by Apple Computer, Inc. for use by Macintosh computers. It is suitable for representing text in English and several other Western languages. Mac OS Roman encodes 256 characters, the first 128 of which ...
* KOI8-R
KOI8-R (RFC 1489) is an 8-bit character encoding, derived from the KOI-8 encoding by the programmer Andrei Chernov in 1993 and designed to cover Russian, which uses a Cyrillic alphabet. KOI8-R was based on Russian Morse code, which was created ...
, KOI8-U
KOI8-U (RFC 2319) is an 8-bit character encoding, designed to cover Ukrainian, which uses a Cyrillic alphabet. It is based on KOI8-R, which covers Russian and Bulgarian, but replaces eight box drawing characters with four Ukrainian letters Ґ ...
, KOI7
* MIK
* ISCII
Indian Script Code for Information Interchange (ISCII) is a coding scheme for representing various writing systems of India. It encodes the main Indic scripts and a Roman transliteration. The supported scripts are: Bengali–Assamese, Devanagar ...
* TSCII
* VISCII
VISCII is an unofficially-defined modified ASCII character encoding for using the Vietnamese language with computers. It should not be confused with the similarly-named officially registered VSCII encoding. VISCII keeps the 95 printable chara ...
* JIS X 0208
JIS X 0208 is a 2-byte character set specified as a Japanese Industrial Standards, Japanese Industrial Standard, containing 6879 graphic characters suitable for writing text, place names, personal names, and so forth in the Japanese language. Th ...
is a widely deployed standard for Japanese character encoding that has several encoding forms.
** Shift JIS
Shift JIS (Shift Japanese Industrial Standards, also SJIS, MIME name Shift_JIS, known as PCK in Solaris contexts) is a character encoding for the Japanese language, originally developed by a Japanese company called ASCII Corporation in conjuncti ...
(Microsoft Code page 932 is a dialect of Shift_JIS)
** EUC-JP
Extended Unix Code (EUC) is a multibyte character encoding system used primarily for Japanese, Korean, and simplified Chinese.
The most commonly used EUC codes are variable-length encodings with a character belonging to an compliant coded char ...
** ISO-2022-JP
ISO/IEC 2022 ''Information technology—Character code structure and extension techniques'', is an ISO/ IEC standard (equivalent to the ECMA standard ECMA-35, the ANSI standard ANSI X3.41 and the Japanese Industrial Standard JIS X 0202) in the ...
* JIS X 0213
JIS X 0213 is a Japanese Industrial Standard defining coded character sets for encoding the characters used in Japan. This standard extends JIS X 0208. The first version was published in 2000 and revised in 2004 (JIS2004) and 2012. As well as a ...
is an extended version of JIS X 0208.
** Shift_JIS-2004
** EUC-JIS-2004
Extended Unix Code (EUC) is a multibyte character encoding system used primarily for Japanese, Korean, and simplified Chinese.
The most commonly used EUC codes are variable-length encodings with a character belonging to an compliant coded charac ...
** ISO-2022-JP-2004
* Chinese Guobiao
The National Standards of the People's Republic of China (), coded as , are the standards issued by the Standardization Administration of China under the authorization of Article 10 of the Standardization Law of the People's Republic of China.
...
** GB 2312
is a key official character set of the People's Republic of China, used for Simplified Chinese characters. GB2312 is the registered internet name for EUC-CN, which is its usual encoded form. ''GB'' refers to the Guobiao standards (国家标准 ...
** GBK (Microsoft Code page 936)
** GB 18030
GB 18030 is a Chinese government standard, described as ''Information Technology — Chinese coded character set'' and defines the required language and character support necessary for software in China. GB18030 is the registered Internet n ...
* Taiwan Big5
Big-5 or Big5 is a Chinese character encoding method used in Taiwan, Hong Kong, and Macau for traditional Chinese characters.
The People's Republic of China (PRC), which uses simplified Chinese characters, uses the GB 18030 character set inst ...
(a more famous variant is Microsoft Code page 950
Code page 950 is the code page used on Microsoft Windows for Traditional Chinese. It is Microsoft's implementation of the ''de facto'' standard Big5 character encoding. The code page is not registered with IANA, and hence, it is not a standard ...
)
** Hong Kong HKSCS
The Hong Kong Supplementary Character Set (; commonly abbreviated to HKSCS) is a set of Chinese characters – 4,702 in total in the initial release—used in Cantonese, as well as when writing the names of some places in Hong Kong (whether in w ...
* Korean
** KS X 1001
KS X 1001, "''Code for Information Interchange (Hangul and Hanja)''", formerly called KS C 5601, is a South Korean coded character set standard to represent hangul and hanja characters on a computer.
KS X 1001 is encoded by the most common le ...
is a Korean double-byte character encoding standard
** EUC-KR
Extended Unix Code (EUC) is a multibyte character encoding system used primarily for Japanese, Korean, and simplified Chinese.
The most commonly used EUC codes are variable-length encodings with a character belonging to an compliant coded char ...
** ISO-2022-KR
ISO/IEC 2022 ''Information technology—Character code structure and extension techniques'', is an ISO/ IEC standard (equivalent to the ECMA standard ECMA-35, the ANSI standard ANSI X3.41 and the Japanese Industrial Standard JIS X 0202) in the ...
* Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, wh ...
(and subsets thereof, such as the 16-bit 'Basic Multilingual Plane')
** UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of ...
** UTF-16
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as cod ...
** UTF-32
UTF-32 (32- bit Unicode Transformation Format) is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode ...
* ANSEL
ANSEL, the American National Standard for Extended Latin Alphabet Coded Character Set for Bibliographic Use, was a character set used in text encoding. It provided a table of coded values for the representation of characters of the extended Latin ...
or ISO/IEC 6937
T.51 / ISO/IEC 6937:2001, ''Information technology — Coded graphic character set for text communication — Latin alphabet'', is a multibyte extension of ASCII, or rather of ISO/IEC 646-IRV. It was developed in common with ITU-T (then CCITT) fo ...
References
Further reading
*
External links
Character sets registered by Internet Assigned Numbers Authority (IANA)
Characters and encodings
by Jukka Korpela
Unicode Technical Report #17: Character Encoding Model
*[https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)] by Joel Spolsky (Oct 10, 2003)
{{DEFAULTSORT:Character Encoding
Character encoding,
Natural language and computing, Encoding