Letter frequency is the number of times
letters
Letter, letters, or literature may refer to:
Characters typeface
* Letter (alphabet), a character representing one or more of the sounds used in speech; any of the symbols of an alphabet.
* Letterform, the graphic form of a letter of the alphabe ...
of the
alphabet
An alphabet is a standardized set of basic written graphemes (called letters) that represent the phonemes of certain spoken languages. Not all writing systems represent language in this way; in a syllabary, each character represents a syllab ...
appear on average in
written language
A written language is the representation of a spoken or gestural language by means of a writing system. Written language is an invention in that it must be taught to children, who will pick up spoken language or sign language by exposure eve ...
. Letter frequency analysis dates back to the
Arab
The Arabs (singular: Arab; singular ar, عَرَبِيٌّ, DIN 31635: , , plural ar, عَرَب, DIN 31635: , Arabic pronunciation: ), also known as the Arab people, are an ethnic group mainly inhabiting the Arab world in Western Asia, ...
mathematician
Al-Kindi
Abū Yūsuf Yaʻqūb ibn ʼIsḥāq aṣ-Ṣabbāḥ al-Kindī (; ar, أبو يوسف يعقوب بن إسحاق الصبّاح الكندي; la, Alkindus; c. 801–873 AD) was an Arab Muslim philosopher, polymath, mathematician, physician ...
(c. 801–873 AD), who formally developed the method to break
ciphers
In cryptography, a cipher (or cypher) is an algorithm for performing encryption or decryption—a series of well-defined steps that can be followed as a procedure. An alternative, less common term is ''encipherment''. To encipher or encode i ...
. Letter frequency analysis gained importance in
Europe
Europe is a large peninsula conventionally considered a continent in its own right because of its great physical size and the weight of its history and traditions. Europe is also considered a subcontinent of Eurasia and it is located entirel ...
with the development of
movable type
Movable type (US English; moveable type in British English) is the system and technology of printing and typography that uses movable components to reproduce the elements of a document (usually individual alphanumeric characters or punctuation m ...
in 1450 AD, where one must estimate the amount of type required for each
letterform
A letterform, letter-form or letter form, is a term used especially in typography, palaeography, calligraphy and epigraphy to mean a letter (alphabet), letter's shape. A letterform is a type of glyph, which is a specific, concrete way of writing ...
. Linguists use letter frequency analysis as a rudimentary technique for
language identification
In natural language processing, language identification or language guessing is the problem of determining which natural language given content is in. Computational approaches to this problem view it as a special case of text categorization, solv ...
, where it is particularly effective as an indication of whether an unknown
writing system
A writing system is a method of visually representing verbal communication, based on a script and a set of rules regulating its use. While both writing and speech are useful in conveying messages, writing differs in also being a reliable fo ...
is alphabetic,
syllabic
Syllabic may refer to:
*Syllable, a unit of speech sound, considered the building block of words
**Syllabic consonant, a consonant that forms the nucleus of a syllable
*Syllabary, writing system using symbols for syllables
*Abugida, writing system ...
, or
ideographic
An ideogram or ideograph (from Greek "idea" and "to write") is a graphic symbol that represents an idea or concept, independent of any particular language, and specific words or phrases. Some ideograms are comprehensible only by familiari ...
.
The use of letter frequencies and
frequency analysis plays a fundamental role in
cryptograms
A cryptogram is a type of puzzle that consists of a short piece of encrypted text. Generally the cipher used to encrypt the text is simple enough that the cryptogram can be solved by hand. Substitution ciphers where each letter is replaced by ...
and several word puzzle games, including
Hangman, ''
Scrabble
''Scrabble'' is a word game in which two to four players score points by placing tiles, each bearing a single letter, onto a game board divided into a 15×15 grid of squares. The tiles must form words that, in crossword fashion, read left t ...
'', ''
Wordle
''Wordle'' is a web-based word game created and developed by Welsh software engineer Josh Wardle, and owned and published by The New York Times Company since 2022. Players have six attempts to guess a five-letter word, with feedback given for ...
'' and the television game show ''
Wheel of Fortune The Wheel of Fortune or ''Rota Fortunae'' has been a concept and metaphor since ancient times referring to the capricious nature of Fate. Wheel of Fortune may also refer to:
Arts, entertainment, and media Art
* ''The Wheel of Fortune'' (Burne-Jo ...
''. One of the earliest descriptions in classical literature of applying the knowledge of English letter frequency to solving a cryptogram is found in
Edgar Allan Poe
Edgar Allan Poe (; Edgar Poe; January 19, 1809 – October 7, 1849) was an American writer, poet, editor, and literary critic. Poe is best known for his poetry and short stories, particularly his tales of mystery and the macabre. He is wid ...
's famous story ''
The Gold-Bug
"The Gold-Bug" is a short story by American writer Edgar Allan Poe published in 1843. The plot follows William Legrand, who was bitten by a gold-colored bug. His servant Jupiter fears that Legrand is going insane and goes to Legrand's friend, an ...
'', where the method is successfully applied to decipher a message giving the location of a treasure hidden by
Captain Kidd
William Kidd, also known as Captain William Kidd or simply Captain Kidd ( – 23 May 1701), was a Scottish sea captain who was commissioned as a privateer and had experience as a pirate. He was tried and executed in London in 1701 for murder a ...
.
Herbert S. Zim
Herbert Spencer Zim (July 12, 1909 – December 5, 1994) was a naturalist, author, editor and educator best known as the founder (1945) and editor-in-chief of the Golden Guides series of nature books.
Biography
Zim was born 1909 in New York ...
, in his classic introductory cryptography text "Codes and Secret Writing", gives the English letter frequency sequence as "ETAON RISHD LFCMU GYPWB VKJXZQ", the most common letter pairs as "TH HE AN RE ER IN ON AT ND ST ES EN OF TE ED OR TI HI AS TO", and the most common
doubled letters as "LL EE SS OO TT FF RR NN PP CC". Different ways of counting can produce somewhat different orders.
Letter frequencies also have a strong effect on the design of some
keyboard layout
A keyboard layout is any specific physical, visual or functional arrangement of the keys, legends, or key-meaning associations (respectively) of a computer keyboard, mobile phone, or other computer-controlled typographic keyboard.
is the actua ...
s. The most frequent letters are on the bottom row of the
Blickensderfer typewriter
The Blickensderfer Typewriter was invented by George Canfield Blickensderfer (1850–1917) and patented on August 4, 1891. Blickensderfer was the nephew of John Celivergos Zachos the inventor of the stenotype. Two models were initially un ...
, and the
home row
Touch typing (also called blind typing, or touch keyboarding) is a style of typing. Although the phrase refers to typing without using the sense of sight to find the keys—specifically, a touch typist will know their location on the keyboard thr ...
of the
Dvorak keyboard layout
Dvorak is a keyboard layout for English patented in 1936 by August Dvorak and his brother-in-law, William Dealey, as a faster and more ergonomic alternative to the QWERTY layout (the ''de facto'' standard keyboard layout). Dvorak proponent ...
.
Background
![CaliforniaJobCaseLayout](https://upload.wikimedia.org/wikipedia/commons/4/4c/CaliforniaJobCaseLayout.png)
The frequency of letters in text has been studied for use in
cryptanalysis, and
frequency analysis in particular, dating back to the Arab mathematician Al-Kindi (c. 801–873 AD), who formally developed the method (the ciphers breakable by this technique go back at least to the
Caesar cipher invented by
Julius Caesar
Gaius Julius Caesar (; ; 12 July 100 BC – 15 March 44 BC), was a Roman general and statesman. A member of the First Triumvirate, Caesar led the Roman armies in the Gallic Wars before defeating his political rival Pompey in a civil war, ...
, so this method could have been explored in classical times). Letter frequency analysis gained additional importance in Europe with the development of movable type in 1450 AD, where one must estimate the amount of type required for each letterform, as evidenced by the variations in letter compartment size in typographer's type cases.
No exact letter frequency distribution underlies a given language, since all writers write slightly differently. However, most languages have a characteristic distribution which is strongly apparent in longer texts. Even language changes as extreme as from
Old English
Old English (, ), or Anglo-Saxon, is the earliest recorded form of the English language, spoken in England and southern and eastern Scotland in the early Middle Ages. It was brought to Great Britain by Anglo-Saxon settlers in the mid-5th c ...
to modern English (regarded as mutually unintelligible) show strong trends in related letter frequencies: over a small sample of Biblical passages, from most frequent to least frequent, of Old English compares to of modern English, with the most extreme differences concerning letterforms not shared.
Linotype machine
The Linotype machine ( ) is a "line casting" machine used in printing; manufactured and sold by the former Mergenthaler Linotype Company and related It was a hot metal typesetting system that cast lines of metal type for individual uses. Lin ...
s for the English language assumed the letter order, from most to least common, to be based on the experience and custom of manual compositors. The equivalent for the French language was .
Arranging the alphabet in Morse into groups of letters that require equal amounts of time to transmit, and then sorting these groups in increasing order, yields .
American Morse code
American Morse Code — also known as Railroad Morse—is the latter-day name for the original version of the Morse Code developed in the mid-1840s, by Samuel Morse and Alfred Vail for their electric telegraph. The "American" qualifier was added ...
was developed in the 1830s by Alfred Vail
Alfred Lewis Vail (September 25, 1807 – January 18, 1859) was an American machinist and inventor. Along with Samuel Morse, Vail was central in developing and commercializing American telegraphy between 1837 and 1844.
Vail and Morse were the f ...
, based on English-language letter frequencies, to encode the most frequent letters with the shortest symbols. Some efficiency was lost in the reformed version now used: the International Morse Code. Letter frequency was used by other telegraph systems, such as the
Murray Code
The Baudot code is an early character encoding for telegraphy invented by Émile Baudot in the 1870s. It was the predecessor to the International Telegraph Alphabet No. 2 (ITA2), the most common teleprinter code in use until the advent of ASCII. ...
.
Similar ideas are used in modern
data-compression techniques such as
Huffman coding
In computer science and information theory, a Huffman code is a particular type of optimal prefix code that is commonly used for lossless data compression. The process of finding or using such a code proceeds by means of Huffman coding, an algo ...
.
Letter frequencies, like
word frequencies
A word list (or ''lexicon'') is a list of a language's lexicon (generally sorted by frequency of occurrence either by levels or as a ranked list) within some given text corpus, serving the purpose of vocabulary acquisition. A lexicon sorted by ...
, tend to vary, both by writer and by subject. One cannot write an essay about x-rays without using frequently, and the essay will have an idiosyncratic letter frequency if the essay is about the use of x-rays to treat zebras in
Qatar
Qatar (, ; ar, قطر, Qaṭar ; local vernacular pronunciation: ), officially the State of Qatar,) is a country in Western Asia. It occupies the Qatar Peninsula on the northeastern coast of the Arabian Peninsula in the Middle East; it ...
. Different authors have habits which can be reflected in their use of letters.
Hemingway
Ernest Miller Hemingway (July 21, 1899 – July 2, 1961) was an American novelist, short-story writer, and journalist. His economical and understated style—which he termed the iceberg theory—had a strong influence on 20th-century fi ...
's writing style, for example, is visibly different from
Faulkner
William Cuthbert Faulkner (; September 25, 1897 – July 6, 1962) was an American writer known for his novels and short stories set in the fictional Yoknapatawpha County, based on Lafayette County, Mississippi, where Faulkner spent most of ...
's. Letter,
bigram
A bigram or digram is a sequence of two adjacent elements from a string of tokens, which are typically letters, syllables, or words. A bigram is an ''n''-gram for ''n''=2. The frequency distribution of every bigram in a string is commonly used f ...
,
trigram
Trigrams are a special case of the ''n''-gram, where ''n'' is 3. They are often used in natural language processing for performing statistical analysis of texts and in cryptography for control and use of ciphers and codes.
Frequency
Contex ...
, word frequencies, word length, and sentence length can be calculated for specific authors, and used to prove or disprove authorship of texts, even for authors whose styles are not so divergent.
Accurate average letter frequencies can only be gleaned by analyzing a large amount of representative text. With the availability of modern computing and collections of large
text corpora
In linguistics, a corpus (plural ''corpora'') or text corpus is a language resource consisting of a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical ...
, such calculations are easily made. Examples can be drawn from a variety of sources (press reporting, religious texts, scientific texts and general fiction) and there are differences especially for general fiction with the position of and , with becoming more common.
Also, to note that different dialects of a language will also affect a letter's frequency. For example, an author in the United States would produce something in which is more common than an author in the United Kingdom writing on the same topic: words like "analyze", "apologize", and "recognize" contain the letter in American English, whereas the same words are spelled "analyse", "apologise", and "recognise" in British English. This would highly affect the frequency of the letter as it is a rarely used letter by British speakers in the English language.
The "top twelve" letters constitute about 80% of the total usage. The "top eight" letters constitute about 65% of the total usage. Letter frequency as a function of rank can be fitted well by several rank functions, with the two-parameter
Cocho/Beta rank function being the best. Another rank function with no adjustable free parameter also fits the letter frequency distribution reasonably well (the same function has been used to fit the amino acid frequency in protein sequences.) A spy using the
VIC cipher
Vic (; es, Vic or Pancracio Celdrán (2004). Diccionario de topónimos españoles y sus gentilicios (5ª edición). Madrid: Espasa Calpe. p. 843. ISBN 978-84-670-3054-9. «Vic o Vich (viquense, vigitano, vigatán, ausense, ausetano, ausonense): ...
or some other cipher based on a straddling checkerboard typically uses a mnemonic such as "a sin to err" (dropping the second "r") or "at one sir" to remember the top eight characters.
Relative frequencies of letters in the English language
![Worn keyboard of English speaker](https://upload.wikimedia.org/wikipedia/commons/3/30/Worn_keyboard_of_English_speaker.jpg)
There are three ways to count letter frequency that result in very different charts for common letters. The first method, used in the chart below, is to count letter frequency in root words of a dictionary. The second is to include all word variants when counting, such as "abstracts", "abstracted" and "abstracting" and not just the root word of "abstract". This system results in letters like appearing much more frequently, such as when counting letters from lists of the most used English words on the Internet. A final variant is to count letters based on their frequency of use in actual texts, resulting in certain letter combinations like becoming more common due to the frequent use of common words like "the", "then", "both", "this", etc. Absolute usage frequency measures like this are used when creating keyboard layouts or letter frequencies in old fashioned printing presses.
An analysis of entries in the Concise Oxford dictionary, ignoring frequency of word use, gives an order of "EARIOTNSLCUDPMHGBFYWKVXZJQ".
The letter-frequency table below is taken from Pavel Mička's website, which cites Robert Lewand's ''Cryptological Mathematics''.
According to Lewand, arranged from most to least common in appearance, the letters are: etaoinshrdlcumwfgypbvkjxqz. Lewand's ordering differs slightly from others, such as Cornell University Math Explorer's Project, which produced a table after measuring 40,000 words.
In English, the space is slightly more frequent than the top letter () and the non-alphabetic characters (digits, punctuation, etc.) collectively occupy the fourth position (having already included the space) between and .
Relative frequencies of the first letters of a word in English language
The frequency of the first letters of words or names is helpful in pre-assigning space in physical files and indexes.
Given 26
filing cabinet
A filing cabinet (or sometimes file cabinet in American English) is a piece of office furniture for storing paper documents in file folders. In the most simple context, it is an enclosure for drawers in which items are stored. The two most comm ...
drawers, rather than a 1:1 assignment of one drawer to one letter of the alphabet, it is often useful to use a more equal-frequency-letter code by assigning several low-frequency letters to the same drawer (often one drawer is labeled VWXYZ), and to split up the most-frequent initial letters () into several drawers (often 6 drawers Aa-An, Ao-Az, Ca-Cj, Ck-Cz, Sa-Si, Sj-Sz). The same system is used in some multi-volume works such as some
encyclopedias.
Cutter numbers, another mapping of names to a more equal-frequency code, are used in some libraries.
Both the overall letter distribution and the word-initial letter distribution approximately match the
Zipf distribution and even more closely match the
Yule distribution
Yule, actually Yuletide ("Yule time") is a festival observed by the historical Germanic peoples, later undergoing Christianised reformulation resulting in the now better-known Christmastide. The earliest references to Yule are by way of indig ...
.
Often the frequency distribution of the first digit in each datum is significantly different from the overall frequency of all the digits in a set of numeric data, see
Benford's law
Benford's law, also known as the Newcomb–Benford law, the law of anomalous numbers, or the first-digit law, is an observation that in many real-life sets of numerical data, the leading digit is likely to be small.Arno Berger and Theodore ...
for details.
An analysis by
Peter Norvig
Peter Norvig (born December 14, 1956) is an American computer scientist and Distinguished Education Fellow at the Stanford Institute for Human-Centered AI. He previously served as a director of research and search quality at Google. Norvig is t ...
on Google Books data determined, among other things, the frequency of first letters of English words.
A June 2012 analysis using a text document containing all words in the English language exactly once, found to be the most common starting letter for words in the English language, followed by .
Relative frequencies of letters in other languages
*See
İ and
dotless I
I, or ı, called dotless I, is a letter used in the Latin-script alphabets of Azerbaijani, Crimean Tatar, Gagauz, Kazakh, Tatar, Kyrgyz, and Turkish. It commonly represents the close back unrounded vowel , except in Kazakh where it represen ...
.
The figure below illustrates the frequency distributions of the 26 most common Latin letters across some languages. All of these languages use a similar 25+ character alphabet.
Based on these tables, the '
etaoin shrdlu'-equivalent results for each language is as follows:
*French: ''; (Indo-European: Italic; traditionally, 'esartinulop' is used, in part for its ease of pronunciation
[Perec, Georges; ''Alphabets''; Éditions Galilée, 1976])
*Spanish: ''; (Indo-European: Italic)
*Portuguese: '' (Indo-European: Italic)
*Italian: ''; (Indo-European: Italic)
*Esperanto: '' (artificial language – lexic influenced by Indo-European languages, Romance, Germanic mostly)
*German: ''; (Indo-European: Germanic)
*Swedish: ''; (Indo-European: Germanic)
*Turkish: ''; (Turkic)
*Dutch: ''; (Indo-European: Germanic)
[
*Polish: ''; (Indo-European: Balto-Slavic)
*Danish: ''; (Indo-European: Germanic)
*Icelandic: ''; (Indo-European: Germanic)
*Finnish: ''; (Uralic: Finnic)
*Czech: ''; (Indo-European: Balto-Slavic)
]
See also
* Arabic letter frequency
* Corpus linguistics
*Dvorak keyboard layout
Dvorak is a keyboard layout for English patented in 1936 by August Dvorak and his brother-in-law, William Dealey, as a faster and more ergonomic alternative to the QWERTY layout (the ''de facto'' standard keyboard layout). Dvorak proponent ...
* English word frequency
* Etaoin shrdlu
* RSTLNE (''Wheel of Fortune'')
Explanatory notes
References
External links
*
*
*
*
Letter frequency
simia.net
Useful tables
Useful tables for single letter, digram, trigram, tetragram, and pentagram frequencies based on 20,000 words that take into account word-length and letter-position combinations for words 3 to 7 letters in length:
*
*
*
* {{cite journal , last1=Mayzner , first1=M.S. , last2=Tresselt , first2=M.E. , last3=Wolin , first3=B.R. , title=Tables of pentagram frequency counts for various word-length and letter-position combinations , journal=Psychonomic Monograph Supplements , volume=1 , issue=5 , pages=144–190 , year=1965
Cryptography
Frequency distribution
Grammatology
Phonology
Quantitative linguistics