Microsoft
Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washi ...
was one of the first companies to implement
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, ...
in their products.
Windows NT
Windows NT is a proprietary graphical operating system produced by Microsoft, the first version of which was released on July 27, 1993. It is a processor-independent, multiprocessing and multi-user operating system.
The first version of Win ...
was the first operating system that used "wide characters" in
system call
In computing, a system call (commonly abbreviated to syscall) is the programmatic way in which a computer program requests a service from the operating system on which it is executed. This may include hardware-related services (for example, acc ...
s. Using the (now obsolete)
UCS-2
The Universal Coded Character Set (UCS, Unicode) is a standard set of characters defined by the international standard ISO/IEC 10646, ''Information technology — Universal Coded Character Set (UCS)'' (plus amendments to that standard), w ...
encoding scheme at first, it was upgraded to the
variable-width encoding
A variable-width encoding is a type of character encoding scheme in which codes of differing lengths are used to encode a character set (a repertoire of symbols) for representation, usually in a computer. Most common variable-width encodings a ...
UTF-16
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as cod ...
starting with
Windows 2000
Windows 2000 is a major release of the Windows NT operating system developed by Microsoft and oriented towards businesses. It was the direct successor to Windows NT 4.0, and was released to manufacturing on December 15, 1999, and was offici ...
, allowing a representation of additional planes with surrogate pairs. However Microsoft did not support
UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of e ...
in its API until May 2019, though it now appears to be encouraging its use.
In various Windows families
Windows NT based systems
Current Windows versions and all back to
Windows XP
Windows XP is a major release of Microsoft's Windows NT operating system. It was release to manufacturing, released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct upgrade to its predecessors, Wind ...
and prior
Windows NT
Windows NT is a proprietary graphical operating system produced by Microsoft, the first version of which was released on July 27, 1993. It is a processor-independent, multiprocessing and multi-user operating system.
The first version of Win ...
(3.x, 4.0) are shipped with
system libraries that support string
encoding of two types: 16-bit "Unicode" (
UTF-16
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as cod ...
since
Windows 2000
Windows 2000 is a major release of the Windows NT operating system developed by Microsoft and oriented towards businesses. It was the direct successor to Windows NT 4.0, and was released to manufacturing on December 15, 1999, and was offici ...
) and a (sometimes multibyte) encoding called the "
code page
In computing, a code page is a character encoding and as such it is a specific association of a set of printable characters and control characters with unique numbers. Typically each number represents the binary value in a single byte. (In some c ...
" (or incorrectly referred to as ''
ANSI
The American National Standards Institute (ANSI ) is a private non-profit organization that oversees the development of voluntary consensus standards for products, services, processes, systems, and personnel in the United States. The organi ...
code page''). 16-bit functions have names suffixed with 'W' (from
"wide") such as
SetWindowTextW
. Code page oriented functions use the suffix 'A' for "ANSI" such as
SetWindowTextA
(some other conventions were used for APIs that were copied from other systems, such as
_wfopen/fopen
or
wcslen/strlen
). This split was necessary because many languages, including
C, did not provide a clean way to pass both 8-bit and 16-bit strings to the same function.
'A' functions are implemented as
wrappers that translate the text using the current code page to UTF-16 and then call the corresponding 'W' functions. 'A' functions that return strings do the opposite conversion, turning characters that don't exist in the current locale into '?'.
Microsoft
Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washi ...
attempted to support Unicode "portably" by providing a "UNICODE" switch to the compiler, that switches unsuffixed "generic" calls from the 'A' to the 'W' interface and converts all string constants to "wide" UTF-16 versions. This does not actually work because it does not translate UTF-8 outside of string constants, resulting in code that attempts to open files just not compiling.
Earlier, and independent of the "UNICODE" switch, Windows also provided the Multibyte Character Sets (MBCS) API switch. This changes some functions that don't work in MBCS such as
strrev
to an MBCS-aware one such as
_mbsrev
.
Microsoft documentation used the term "Unicode" to mean "not 8-bit encoding".
Windows CE
In (the now discontinued)
Windows CE
Windows Embedded Compact, formerly Windows Embedded CE, Windows Powered and Windows CE, is an operating system subfamily developed by Microsoft as part of its Windows Embedded family of products.
Unlike Windows Embedded Standard, which is base ...
, UTF-16 was used almost exclusively, with the 'A' API mostly missing. A limited set of ANSI API is available in Windows CE 5.0, for use on a reduced set of locales that may be selectively built onto the runtime image.
Windows 9x
In 2001, Microsoft released a special supplement to Microsoft's old
Windows 9x
Windows 9x is a generic term referring to a series of Microsoft Windows computer operating systems produced from 1995 to 2000, which were based on the Windows 95 kernel and its underlying foundation of MS-DOS, both of which were updated in sub ...
systems. It includes a dynamic link library, 'unicows.dll', (only 240 KB) containing the 16-bit flavor (the ones with the letter W on the end) of all the basic functions of Windows API. It is merely a translation layer:
SetWindowTextW
will simply convert its input using the current codepage and call
SetWindowTextA
.
UTF-8
Microsoft Windows (
Windows XP
Windows XP is a major release of Microsoft's Windows NT operating system. It was release to manufacturing, released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct upgrade to its predecessors, Wind ...
and later) has a code page designated for
UTF-8
UTF-8 is a variable-length character encoding used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode'' (or ''Universal Coded Character Set'') ''Transformation Format 8-bit''.
UTF-8 is capable of e ...
, code page 65001 (i.e.
CP_UTF8
). Prior to Windows 10 insider build 17035 (November 2017), it was impossible to set the locale code page to 65001, leaving this code page only available for (a) explicit conversion functions such as MultiByteToWideChar and/or (b) the
Win32 console
Windows Console is the infrastructure for console applications in Microsoft Windows. An instance of a Windows Console has a screen buffer and an input buffer. It allows console apps to run inside a window or in hardware text mode (so as to oc ...
command
chcp 65001
to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular
fopen
(which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using
fopen
no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as
SetWindowText
.
On all modern non-Windows platforms, the file-name string passed to
fopen
is effectively UTF-8. This produced an incompatibility between other platforms and Windows. The usual work-around was to add code to convert UTF-8 to UTF-16 using
MultiByteToWideChar
Windows code pages are sets of characters or code pages (known as character encodings in other operating systems) used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode was implemented in Wind ...
and call the "wide" function instead of
fopen
. Another popular work-around was to convert the name to the
8.3 filename
An 8.3 filename (also called a short filename or SFN) is a filename convention used by old versions of DOS and versions of Microsoft Windows prior to Windows 95 and Windows NT 3.5. It is also used in modern Microsoft operating systems as an altern ...
equivalent, this is necessary if the
fopen
is inside a library function that takes a string filename and thus calling another function is not possible. There were also proposals to add new APIs to portable libraries such as
Boost to do the necessary conversion, by adding new functions for opening and renaming files. These functions would pass filenames through unchanged on Unix, but translate them to UTF-16 on Windows. Such a library, Boost.Nowide, was accepted into Boost and will be part of the 1.73 release. This would allow code to be "portable", but required just as many code changes as calling the wide functions.
In April 2018, with insider build 17035 (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8. This allows for calling "narrow" functions, including
fopen
and
SetWindowTextA
, with UTF-8 strings.
In May 2019, Microsoft added the ability for a program to set the code page to UTF-8 itself,
allowing programs written to use UTF-8 to be run by non-expert users.
In
Windows 11
Windows 11 is the latest major release of Microsoft's Windows NT operating system, released in October 2021. It is a free upgrade to its predecessor, Windows 10 (2015), and is available for any Windows 10 devices that meet the new Windows 11 ...
some system files are required to use UTF-8 and do not require a Byte Order Mark. Notepad can now recognize UTF-8 without the Byte Order Mark, and can be told to write UTF-8 without a Byte Order Mark. Some other Microsoft products are using UTF-8 internally, including Visual Studio and their
SQL Server 2019, with Microsoft claiming 35% speed increase from use of UTF-8, and "nearly 50% reduction in storage requirements."
Programming platforms
Microsoft's compilers often fail at producing UTF-8 string constants from UTF-8 source files. The most reliable method is to turn ''off'' UNICODE, ''not'' mark the input file as being UTF-8 (i.e. do not use a
BOM), and arrange the string constants to have the UTF-8 bytes. If a BOM was added, a Microsoft compiler will interpret the strings as UTF-8, convert them to UTF-16, then convert them ''back'' into the current locale, thus destroying the UTF-8.
UTF-8 Everywhere FAQ: How do I write UTF-8 string literal in my C++ code?
/ref> Without a BOM and using a single-byte locale, Microsoft compilers will leave the bytes in a quoted string unchanged. On modern systems setting the code page to UTF-8 helps considerably, but invalid byte sequences are still not preserved (using can work around this).
See also
* Bush hid the facts
Bush hid the facts is a common name for a bug present in some versions of Microsoft Windows, which causes text encoded in ASCII to be interpreted as if it were UTF-16LE, resulting in garbled text. When the string "Bush hid the facts", without q ...
, a text encoding mojibake
Mojibake ( ja, 文字化け; , "character transformation") is the garbled text that is the result of text being decoded using an unintended character encoding. The result is a systematic replacement of symbols with completely unrelated ones, oft ...
Notes
References
External links
* {{cite web , url=http://msdn.microsoft.com/en-us/library/dd374081(v=vs.85).aspx , title=Unicode , work=MSDN
Microsoft Developer Network (MSDN) was the division of Microsoft responsible for managing the firm's relationship with developers and testers, such as hardware developers interested in the operating system (OS), and software developers developing ...
, publisher=Microsoft
Microsoft Corporation is an American multinational technology corporation producing computer software, consumer electronics, personal computers, and related services headquartered at the Microsoft Redmond campus located in Redmond, Washi ...
, access-date=November 10, 2016
Unicode
Unicode, formally The Unicode Standard,The formal version reference is is an information technology standard for the consistent encoding, representation, and handling of text expressed in most of the world's writing systems. The standard, ...
Microsoft Windows
Windows is a group of several proprietary graphical operating system families developed and marketed by Microsoft. Each family caters to a certain sector of the computing industry. For example, Windows NT for consumers, Windows Server for ...