Speech recognition is an

interdisciplinary Interdisciplinarity or interdisciplinary studies involves the combination of multiple academic disciplines into one activity (e.g., a research project). It draws knowledge from several other fields like sociology, anthropology, psychology, ec ...

subfield of

computer science Computer science is the study of computation, automation, and information. Computer science spans theoretical disciplines (such as algorithms, theory of computation, information theory, and automation) to Applied science, practical discipli ...

and

computational linguistics Computational linguistics is an Interdisciplinarity, interdisciplinary field concerned with the computational modelling of natural language, as well as the study of appropriate computational approaches to linguistic questions. In general, comput ...

that develops methodologies and technologies that enable the recognition and

translation Translation is the communication of the Meaning (linguistic), meaning of a #Source and target languages, source-language text by means of an Dynamic and formal equivalence, equivalent #Source and target languages, target-language text. The ...

of spoken language into text by computers with the main benefit of searchability. It is also known as automatic speech recognition (ASR), computer speech recognition or speech to text (STT). It incorporates knowledge and research in the

linguistics Linguistics is the scientific study of human language. It is called a scientific study because it entails a comprehensive, systematic, objective, and precise analysis of all aspects of language, particularly its nature and structure. Linguis ...

and

computer engineering Computer engineering (CoE or CpE) is a branch of electrical engineering and computer science that integrates several fields of computer science and electronic engineering required to develop computer hardware and software. Computer engineers ...

fields. The reverse process is

speech synthesis Speech synthesis is the artificial production of human speech. A computer system used for this purpose is called a speech synthesizer, and can be implemented in software or hardware products. A text-to-speech (TTS) system converts normal languag ...

. Some speech recognition systems require "training" (also called "enrollment") where an individual speaker reads text or isolated

vocabulary A vocabulary is a set of familiar words within a person's language. A vocabulary, usually developed with age, serves as a useful and fundamental tool for communication and acquiring knowledge. Acquiring an extensive vocabulary is one of the la ...

into the system. The system analyzes the person's specific voice and uses it to fine-tune the recognition of that person's speech, resulting in increased accuracy. Systems that do not use training are called "speaker-independent" systems. Systems that use training are called "speaker dependent". Speech recognition applications include

voice user interface A voice-user interface (VUI) makes spoken human interaction with computers possible, using speech recognition to understand spoken commands and answer questions, and typically text to speech to play a reply. A voice command device is a device con ...

s such as voice dialing (e.g. "call home"), call routing (e.g. "I would like to make a collect call"),

domotic Home automation or domotics is building automation for a home, called a smart home or smart house. A home automation system will monitor and/or control home attributes such as lighting, climate, entertainment systems, and appliances. It m ...

appliance control, search key words (e.g. find a podcast where particular words were spoken), simple data entry (e.g., entering a credit card number), preparation of structured documents (e.g. a radiology report), determining speaker characteristics, speech-to-text processing (e.g.,

word processor A word processor (WP) is a device or computer program that provides for input, editing, formatting, and output of text, often with some additional features. Word processor (electronic device), Early word processors were stand-alone devices ded ...

s or

email Electronic mail (email or e-mail) is a method of exchanging messages ("mail") between people using electronic devices. Email was thus conceived as the electronic ( digital) version of, or counterpart to, mail, at a time when "mail" meant ...

s), and

aircraft An aircraft is a vehicle that is able to fly by gaining support from the air. It counters the force of gravity by using either static lift or by using the dynamic lift of an airfoil, or in a few cases the downward thrust from jet engines ...

(usually termed

direct voice input Direct voice input (DVI), sometimes called voice input control (VIC), is a style of human–machine interaction "HMI" in which the user makes voice commands to issue instructions to the machine through speech recognition. In the field of militar ...

). The term ''voice recognition'' or ''

speaker identification Speaker recognition is the identification of a person from characteristics of voices. It is used to answer the question "Who is speaking?" The term voice recognition can refer to ''speaker recognition'' or speech recognition. Speaker verification ...

'' refers to identifying the speaker, rather than what they are saying. Recognizing the speaker can simplify the task of translating speech in systems that have been trained on a specific person's voice or it can be used to authenticate or verify the identity of a speaker as part of a security process. From the technology perspective, speech recognition has a long history with several waves of major innovations. Most recently, the field has benefited from advances in

deep learning Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised. De ...

and

big data Though used sometimes loosely partly because of a lack of formal definition, the interpretation that seems to best describe Big data is the one associated with large body of information that we could not comprehend when used only in smaller am ...

. The advances are evidenced not only by the surge of academic papers published in the field, but more importantly by the worldwide industry adoption of a variety of deep learning methods in designing and deploying speech recognition systems.

History

The key areas of growth were: vocabulary size, speaker independence, and processing speed.

Pre-1970

* 1952 – Three Bell Labs researchers, Stephen Balashek, R. Biddulph, and K. H. Davis built a system called "Audrey" for single-speaker digit recognition. Their system located the

formants In speech science and phonetics, a formant is the broad spectral maximum that results from an acoustic resonance of the human vocal tract. In acoustics, a formant is usually defined as a broad peak, or local maximum, in the spectrum. For harmonic ...

in the power spectrum of each utterance. * 1960 – Gunnar Fant developed and published the source-filter model of speech production. * 1962 – IBM demonstrated its 16-word "Shoebox" machine's speech recognition capability at the

1962 World's Fair The Century 21 Exposition (also known as the Seattle World's Fair) was a world's fair held April 21, 1962, to October 21, 1962, in Seattle, Washington (state), Washington, United States.Linear predictive coding (LPC), a speech coding method, was first proposed by Fumitada Itakura of

Nagoya University , abbreviated to or NU, is a Japanese national research university located in Chikusa-ku, Nagoya. It was the seventh Imperial University in Japan, one of the first five Designated National University and selected as a Top Type university of T ...

and Shuzo Saito of Nippon Telegraph and Telephone (NTT), while working on speech recognition. * 1969 – Funding at

Bell Labs Nokia Bell Labs, originally named Bell Telephone Laboratories (1925–1984), then AT&T Bell Laboratories (1984–1996) and Bell Labs Innovations (1996–2007), is an American industrial research and scientific development company owned by mult ...

dried up for several years when, in 1969, the influential John Pierce wrote an open letter that was critical of and defunded speech recognition research. This defunding lasted until Pierce retired and James L. Flanagan took over.

Raj Reddy Dabbala Rajagopal "Raj" Reddy (born 13 June 1937) is an Indian-American computer scientist and a winner of the Turing Award. He is one of the early pioneers of artificial intelligence and has served on the faculty of Stanford and Carnegie Mello ...

was the first person to take on continuous speech recognition as a graduate student at

Stanford University Stanford University, officially Leland Stanford Junior University, is a private research university in Stanford, California. The campus occupies , among the largest in the United States, and enrolls over 17,000 students. Stanford is consider ...

in the late 1960s. Previous systems required users to pause after each word. Reddy's system issued spoken commands for playing

chess Chess is a board game for two players, called White and Black, each controlling an army of chess pieces in their color, with the objective to checkmate the opponent's king. It is sometimes called international chess or Western chess to disti ...

. Around this time Soviet researchers invented the dynamic time warping (DTW) algorithm and used it to create a recognizer capable of operating on a 200-word vocabulary. DTW processed speech by dividing it into short frames, e.g. 10ms segments, and processing each frame as a single unit. Although DTW would be superseded by later algorithms, the technique carried on. Achieving speaker independence remained unsolved at this time period.

1970–1990

* 1971 –

DARPA The Defense Advanced Research Projects Agency (DARPA) is a research and development agency of the United States Department of Defense responsible for the development of emerging technologies for use by the military. Originally known as the Adv ...

funded five years for ''Speech Understanding Research'', speech recognition research seeking a minimum vocabulary size of 1,000 words. They thought speech ''understanding'' would be key to making progress in speech ''recognition'', but this later proved untrue. BBN, IBM, Carnegie Mellon and Stanford Research Institute all participated in the program. This revived speech recognition research post John Pierce's letter. * 1972 – The IEEE Acoustics, Speech, and Signal Processing group held a conference in Newton, Massachusetts. * 1976 – The first

ICASSP ICASSP, the International Conference on Acoustics, Speech, and Signal Processing, is an annual flagship conference organized of IEEE Signal Processing Society. All papers included in its proceedings have been indexed by Ei Compendex. The first ICAS ...

was held in

Philadelphia Philadelphia, often called Philly, is the largest city in the Commonwealth of Pennsylvania, the sixth-largest city in the U.S., the second-largest city in both the Northeast megalopolis and Mid-Atlantic regions after New York City. Sinc ...

, which since then has been a major venue for the publication of research on speech recognition. During the late 1960s Leonard Baum developed the mathematics of

Markov chain A Markov chain or Markov process is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Informally, this may be thought of as, "What happe ...

s at the Institute for Defense Analysis. A decade later, at CMU, Raj Reddy's students James Baker and

Janet M. Baker Janet MacIver Baker and her husband James K. Baker are the co-founders of Dragon Systems. Together they are credited with creation of Dragon NaturallySpeaking. In 2012, she received the IEEE James L. Flanagan Speech and Audio Processing Award ...

began using the

Hidden Markov Model A hidden Markov model (HMM) is a statistical Markov model in which the system being modeled is assumed to be a Markov process — call it X — with unobservable ("''hidden''") states. As part of the definition, HMM requires that there be an ob ...

(HMM) for speech recognition. James Baker had learned about HMMs from a summer job at the Institute of Defense Analysis during his undergraduate education. The use of HMMs allowed researchers to combine different sources of knowledge, such as acoustics, language, and syntax, in a unified probabilistic model. * By the mid-1980s IBM's Fred Jelinek's team created a voice activated typewriter called Tangora, which could handle a 20,000-word vocabulary Jelinek's statistical approach put less emphasis on emulating the way the human brain processes and understands speech in favor of using statistical modeling techniques like HMMs. (Jelinek's group independently discovered the application of HMMs to speech.) This was controversial with linguists since HMMs are too simplistic to account for many common features of human languages. However, the HMM proved to be a highly useful way for modeling speech and replaced dynamic time warping to become the dominant speech recognition algorithm in the 1980s. * 1982 – Dragon Systems, founded by James and

, was one of IBM's few competitors.

Practical speech recognition

The 1980s also saw the introduction of the

n-gram In the fields of computational linguistics and probability, an ''n''-gram (sometimes also called Q-gram) is a contiguous sequence of ''n'' items from a given sample of text or speech. The items can be phonemes, syllables, letters, words or b ...

language model. * 1987 – The back-off model allowed language models to use multiple length n-grams, and CSELT used HMM to recognize languages (both in software and in hardware specialized processors, e.g. RIPAC). Much of the progress in the field is owed to the rapidly increasing capabilities of computers. At the end of the DARPA program in 1976, the best computer available to researchers was the

PDP-10 Digital Equipment Corporation (DEC)'s PDP-10, later marketed as the DECsystem-10, is a mainframe computer family manufactured beginning in 1966 and discontinued in 1983. 1970s models and beyond were marketed under the DECsystem-10 name, especi ...

with 4 MB ram. It could take up to 100 minutes to decode just 30 seconds of speech. Two practical products were: * 1984 – was released the Apricot Portable with up to 4096 words support, of which only 64 could be held in RAM at a time. *1987 – a recognizer from Kurzweil Applied Intelligence * 1990 – Dragon Dictate, a consumer product released in 1990

AT&T AT&T Inc. is an American multinational telecommunications holding company headquartered at Whitacre Tower in Downtown Dallas, Texas. It is the world's largest telecommunications company by revenue and the third largest provider of mobile tel ...

deployed the Voice Recognition Call Processing service in 1992 to route telephone calls without the use of a human operator. The technology was developed by

Lawrence Rabiner Lawrence R. Rabiner (born 28 September 1943) is an electrical engineer working in the fields of digital signal processing and speech processing; in particular in digital signal processing for automatic speech recognition. He has worked on system ...

and others at Bell Labs. By this point, the vocabulary of the typical commercial speech recognition system was larger than the average human vocabulary. Raj Reddy's former student,

Xuedong Huang Xuedong D. Huang (born October 20, 1962) is a Chinese American computer scientist and technology executive who has made contributions to spoken language processing and AI Cognitive Services. He is Microsoft's Technical Fellow and Chief Technology ...

, developed the Sphinx-II system at CMU. The Sphinx-II system was the first to do speaker-independent, large vocabulary, continuous speech recognition and it had the best performance in DARPA's 1992 evaluation. Handling continuous speech with a large vocabulary was a major milestone in the history of speech recognition. Huang went on to found the speech recognition group at Microsoft in 1993. Raj Reddy's student Kai-Fu Lee joined Apple where, in 1992, he helped develop a speech interface prototype for the Apple computer known as Casper. Lernout & Hauspie, a Belgium-based speech recognition company, acquired several other companies, including Kurzweil Applied Intelligence in 1997 and Dragon Systems in 2000. The L&H speech technology was used in the

Windows XP Windows XP is a major release of Microsoft's Windows NT operating system. It was released to manufacturing on August 24, 2001, and later to retail on October 25, 2001. It is a direct upgrade to its predecessors, Windows 2000 for high-end and ...

operating system. L&H was an industry leader until an accounting scandal brought an end to the company in 2001. The speech technology from L&H was bought by ScanSoft which became Nuance in 2005.

Apple An apple is an edible fruit produced by an apple tree (''Malus domestica''). Apple fruit tree, trees are agriculture, cultivated worldwide and are the most widely grown species in the genus ''Malus''. The tree originated in Central Asia, wh ...

originally licensed software from Nuance to provide speech recognition capability to its digital assistant Siri.

2000s

In the 2000s DARPA sponsored two speech recognition programs: Effective Affordable Reusable Speech-to-Text (EARS) in 2002 and

Global Autonomous Language Exploitation The Global Autonomous Language Exploitation (GALE) program was funded by DARPA starting in 2005 to develop technologies for automatic information extraction from multilingual newscasts, documents and other forms of communication. The program encomp ...

(GALE). Four teams participated in the EARS program: IBM, a team led by BBN with LIMSI and Univ. of Pittsburgh,

Cambridge University , mottoeng = Literal: From here, light and sacred draughts. Non literal: From this place, we gain enlightenment and precious knowledge. , established = , other_name = The Chancellor, Masters and Schola ...

, and a team composed of ICSI, SRI and

University of Washington The University of Washington (UW, simply Washington, or informally U-Dub) is a public research university in Seattle, Washington. Founded in 1861, Washington is one of the oldest universities on the West Coast; it was established in Seattle a ...

. EARS funded the collection of the Switchboard telephone

speech corpus A speech corpus (or spoken corpus) is a database of speech audio files and text transcriptions. In speech technology, speech corpora are used, among other things, to create acoustic models (which can then be used with a speech recognition or spea ...

containing 260 hours of recorded conversations from over 500 speakers. The GALE program focused on

Arabic Arabic (, ' ; , ' or ) is a Semitic languages, Semitic language spoken primarily across the Arab world.Semitic languages: an international handbook / edited by Stefan Weninger; in collaboration with Geoffrey Khan, Michael P. Streck, Janet C ...

and

Mandarin Mandarin or The Mandarin may refer to: Language * Mandarin Chinese, branch of Chinese originally spoken in northern parts of the country ** Standard Chinese or Modern Standard Mandarin, the official language of China ** Taiwanese Mandarin, Stand ...

broadcast news speech.

Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...

's first effort at speech recognition came in 2007 after hiring some researchers from Nuance. The first product was

GOOG-411 GOOG-411 (or Google Voice Local Search) was a telephone service launched by Google in 2007, that provided a speech-recognition-based business directory search, and placed a call to the resulting number in the United States or Canada. The servi ...

, a telephone based directory service. The recordings from GOOG-411 produced valuable data that helped Google improve their recognition systems.

Google Voice Search Google Voice Search or Search by Voice is a Google product that allows users to use Google Search by Voice search, speaking on a mobile phone or computer, i.e. have the device search for data upon entering information on what to search into the ...

is now supported in over 30 languages. In the United States, the

National Security Agency The National Security Agency (NSA) is a national-level intelligence agency of the United States Department of Defense, under the authority of the Director of National Intelligence (DNI). The NSA is responsible for global monitoring, collecti ...

has made use of a type of speech recognition for

keyword spotting Keyword spotting (or more simply, word spotting) is a problem that was historically first defined in the context of speech processing. In speech processing, keyword spotting deals with the identification of keywords in utterances. Keyword spotting ...

since at least 2006. This technology allows analysts to search through large volumes of recorded conversations and isolate mentions of keywords. Recordings can be indexed and analysts can run queries over the database to find conversations of interest. Some government research programs focused on intelligence applications of speech recognition, e.g. DARPA's EARS's program and IARPA's

Babel program The IARPA Babel program developed speech recognition technology for noisy telephone conversations. The main goal of the program was to improve the performance of keyword search on languages with very little transcribed data, i.e. low-resource langua ...

. In the early 2000s, speech recognition was still dominated by traditional approaches such as

Hidden Markov Models A hidden Markov model (HMM) is a statistical Markov model in which the system being modeled is assumed to be a Markov process — call it X — with unobservable ("''hidden''") states. As part of the definition, HMM requires that there be an obs ...

combined with feedforward

artificial neural networks Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected unit ...

.Herve Bourlard and

Nelson Morgan Nelson Harold Morgan (born May, 1949) is an American computer scientist and professor in residence (emeritus) of electrical engineering and computer science at the University of California, Berkeley. Morgan is the co-inventor of the Relative Spec ...

, Connectionist Speech Recognition: A Hybrid Approach, The Kluwer International Series in Engineering and Computer Science; v. 247, Boston: Kluwer Academic Publishers, 1994. Today, however, many aspects of speech recognition have been taken over by a

method called

Long short-term memory Long short-term memory (LSTM) is an artificial neural network used in the fields of artificial intelligence and deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. Such a recurrent neural network (RNN) ca ...

(LSTM), a recurrent neural network published by

Sepp Hochreiter Josef "Sepp" Hochreiter (born 14 February 1967) is a German computer scientist. Since 2018 he has led the Institute for Machine Learning at the Johannes Kepler University of Linz after having led the Institute of Bioinformatics from 2006 to 2018. ...

Jürgen Schmidhuber Jürgen Schmidhuber (born 17 January 1963) is a German computer scientist most noted for his work in the field of artificial intelligence, deep learning and artificial neural networks. He is a co-director of the Dalle Molle Institute for Artif ...

in 1997. LSTM RNNs avoid the vanishing gradient problem and can learn "Very Deep Learning" tasks that require memories of events that happened thousands of discrete time steps ago, which is important for speech. Around 2007, LSTM trained by Connectionist Temporal Classification (CTC)Alex Graves, Santiago Fernandez, Faustino Gomez, and

(2006)
Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural nets
Proceedings of ICML'06, pp. 369–376. started to outperform traditional speech recognition in certain applications.Santiago Fernandez, Alex Graves, and Jürgen Schmidhuber (2007)
An application of recurrent neural networks to discriminative keyword spotting
Proceedings of ICANN (2), pp. 220–229. In 2015, Google's speech recognition reportedly experienced a dramatic performance jump of 49% through CTC-trained LSTM, which is now available through

Google Voice Google Voice is a telephone service that provides a U.S. phone number to Google Account customers in the U.S. and Google Workspace (G Suite by October 2020) customers in Canada, Denmark, France, the Netherlands, Portugal, Spain, Sweden, Switz ...

to all smartphone users.Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays and Johan Schalkwyk (September 2015):
Google voice search: faster and more accurate
" The use of deep feedforward (non-recurrent) networks for acoustic modeling was introduced during the later part of 2009 by Geoffrey Hinton and his students at the University of Toronto and by Li Deng and colleagues at Microsoft Research, initially in the collaborative work between Microsoft and the University of Toronto which was subsequently expanded to include IBM and Google (hence "The shared views of four research groups" subtitle in their 2012 review paper).NIPS Workshop: Deep Learning for Speech Recognition and Related Applications, Whistler, BC, Canada, Dec. 2009 (Organizers: Li Deng, Geoff Hinton, D. Yu). A Microsoft research executive called this innovation "the most dramatic change in accuracy since 1979". In contrast to the steady incremental improvements of the past few decades, the application of deep learning decreased word error rate by 30%. This innovation was quickly adopted across the field. Researchers have begun to use deep learning techniques for language modeling as well. In the long history of speech recognition, both shallow form and deep form (e.g. recurrent nets) of artificial neural networks had been explored for many years during 1980s, 1990s and a few years into the 2000s.Morgan, Bourlard, Renals, Cohen, Franco (1993) "Hybrid neural network/hidden Markov model systems for continuous speech recognition. ICASSP/IJPRAI" Waibel, Hanazawa, Hinton, Shikano, Lang. (1989)
Phoneme recognition using time-delay neural networks
IEEE Transactions on Acoustics, Speech, and Signal Processing." But these methods never won over the non-uniform internal-handcrafting

Gaussian mixture model In statistics, a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observatio ...

Hidden Markov model A hidden Markov model (HMM) is a statistical Markov model in which the system being modeled is assumed to be a Markov process — call it X — with unobservable ("''hidden''") states. As part of the definition, HMM requires that there be an ob ...

(GMM-HMM) technology based on generative models of speech trained discriminatively. A number of key difficulties had been methodologically analyzed in the 1990s, including gradient diminishing

(1991)
Untersuchungen zu dynamischen neuronalen Netzen
, Diploma thesis. Institut f. Informatik, Technische Univ. Munich. Advisor: J. Schmidhuber. and weak temporal correlation structure in the neural predictive models. All these difficulties were in addition to the lack of big training data and big computing power in these early days. Most speech recognition researchers who understood such barriers hence subsequently moved away from neural nets to pursue generative modeling approaches until the recent resurgence of deep learning starting around 2009–2010 that had overcome all these difficulties. Hinton et al. and Deng et al. reviewed part of this recent history about how their collaboration with each other and then with colleagues across four groups (University of Toronto, Microsoft, Google, and IBM) ignited a renaissance of applications of deep feedforward neural networks to speech recognition.Keynote talk: Recent Developments in Deep Neural Networks. ICASSP, 2013 (by Geoff Hinton).Keynote talk:

" Interspeech, September 2014 (by

Li Deng Li, li, or LI may refer to: Businesses and organizations * Landscape Institute, a British professional body for landscape architects * Leadership Institute, a non-profit organization located in Arlington, Virginia, US, that teaches "political tec ...

2010s

By early 2010s ''speech'' recognition, also called voice recognition was clearly differentiated from ''speaker'' recognition, and speaker independence was considered a major breakthrough. Until then, systems required a "training" period. A 1987 ad for a doll had carried the tagline "Finally, the doll that understands you." – despite the fact that it was described as "which children could train to respond to their voice". In 2017, Microsoft researchers reached a historical human parity milestone of transcribing conversational telephony speech on the widely benchmarked Switchboard task. Multiple deep learning models were used to optimize speech recognition accuracy. The speech recognition word error rate was reported to be as low as 4 professional human transcribers working together on the same benchmark, which was funded by IBM Watson speech team on the same task.

Models, methods, and algorithms

Both acoustic modeling and

language model A language model is a probability distribution over sequences of words. Given any sequence of words of length , a language model assigns a probability P(w_1,\ldots,w_m) to the whole sequence. Language models generate probabilities by training on ...

ing are important parts of modern statistically based speech recognition algorithms. Hidden Markov models (HMMs) are widely used in many systems. Language modeling is also used in many other natural language processing applications such as document classification or statistical machine translation.

Hidden Markov models

Modern general-purpose speech recognition systems are based on hidden Markov models. These are statistical models that output a sequence of symbols or quantities. HMMs are used in speech recognition because a speech signal can be viewed as a piecewise stationary signal or a short-time stationary signal. In a short time scale (e.g., 10 milliseconds), speech can be approximated as a stationary process. Speech can be thought of as a Markov model for many stochastic purposes. Another reason why HMMs are popular is that they can be trained automatically and are simple and computationally feasible to use. In speech recognition, the hidden Markov model would output a sequence of ''n''-dimensional real-valued vectors (with ''n'' being a small integer, such as 10), outputting one of these every 10 milliseconds. The vectors would consist of

cepstral In Fourier analysis, the cepstrum (; plural ''cepstra'', adjective ''cepstral'') is the result of computing the inverse Fourier transform (IFT) of the logarithm of the estimated signal spectrum. The method is a tool for investigating periodic str ...

coefficients, which are obtained by taking a

Fourier transform A Fourier transform (FT) is a mathematical transform that decomposes functions into frequency components, which are represented by the output of the transform as a function of frequency. Most commonly functions of time or space are transformed, ...

of a short time window of speech and decorrelating the spectrum using a

cosine transform In mathematics, the Fourier sine and cosine transforms are forms of the Fourier transform that do not use complex numbers or require negative frequency. They are the forms originally used by Joseph Fourier and are still preferred in some application ...

, then taking the first (most significant) coefficients. The hidden Markov model will tend to have in each state a statistical distribution that is a mixture of diagonal covariance Gaussians, which will give a likelihood for each observed vector. Each word, or (for more general speech recognition systems), each

phoneme In phonology and linguistics, a phoneme () is a unit of sound that can distinguish one word from another in a particular language. For example, in most dialects of English, with the notable exception of the West Midlands and the north-west o ...

, will have a different output distribution; a hidden Markov model for a sequence of words or phonemes is made by concatenating the individual trained hidden Markov models for the separate words and phonemes. Described above are the core elements of the most common, HMM-based approach to speech recognition. Modern speech recognition systems use various combinations of a number of standard techniques in order to improve results over the basic approach described above. A typical large-vocabulary system would need

context dependency Context may refer to: * Context (language use), the relevant constraints of the communicative situation that influence language use, language variation, and discourse summary Computing * Context (computing), the virtual environment required to su ...

for the phonemes (so phonemes with different left and right context have different realizations as HMM states); it would use

cepstral normalization In Fourier analysis, the cepstrum (; plural ''cepstra'', adjective ''cepstral'') is the result of computing the inverse Fourier transform (IFT) of the logarithm of the estimated signal spectrum. The method is a tool for investigating periodic stru ...

to normalize for a different speaker and recording conditions; for further speaker normalization, it might use vocal tract length normalization (VTLN) for male-female normalization and

maximum likelihood linear regression In mathematical analysis, the maxima and minima (the respective plurals of maximum and minimum) of a function, known collectively as extrema (the plural of extremum), are the largest and smallest value of the function, either within a given ran ...

(MLLR) for more general speaker adaptation. The features would have so-called

delta Delta commonly refers to: * Delta (letter) (Δ or δ), a letter of the Greek alphabet * River delta, at a river mouth * D (NATO phonetic alphabet: "Delta") * Delta Air Lines, US * Delta variant of SARS-CoV-2 that causes COVID-19 Delta may also re ...

and delta-delta coefficients to capture speech dynamics and in addition, might use

heteroscedastic linear discriminant analysis In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...

(HLDA); or might skip the delta and delta-delta coefficients and use splicing and an

LDA LDA may refer to: Aviation *Localizer type directional aid, an instrument approach to an airport *Landing distance available, the length of runway that is available for the ground run of an airplane landing Law *Legal document assistant, a non-la ...

-based projection followed perhaps by

heteroscedastic In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...

linear discriminant analysis or a

global semi-tied co variance Global means of or referring to a globe and may also refer to: Entertainment * ''Global'' (Paul van Dyk album), 2003 * ''Global'' (Bunji Garlin album), 2007 * ''Global'' (Humanoid album), 1989 * ''Global'' (Todd Rundgren album), 2015 * Bruno ...

transform (also known as

maximum likelihood linear transform In mathematical analysis, the maxima and minima (the respective plurals of maximum and minimum) of a function, known collectively as extrema (the plural of extremum), are the largest and smallest value of the function, either within a given ran ...

, or MLLT). Many systems use so-called discriminative training techniques that dispense with a purely statistical approach to HMM parameter estimation and instead optimize some classification-related measure of the training data. Examples are maximum

mutual information In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the " amount of information" (in units such ...

(MMI), minimum classification error (MCE), and minimum phone error (MPE). Decoding of the speech (the term for what happens when the system is presented with a new utterance and must compute the most likely source sentence) would probably use the Viterbi algorithm to find the best path, and here there is a choice between dynamically creating a combination hidden Markov model, which includes both the acoustic and language model information and combining it statically beforehand (the finite state transducer, or FST, approach). A possible improvement to decoding is to keep a set of good candidates instead of just keeping the best candidate, and to use a better scoring function ( re scoring) to rate these good candidates so that we may pick the best one according to this refined score. The set of candidates can be kept either as a list (the N-best list approach) or as a subset of the models (a lattice). Re scoring is usually done by trying to minimize the

Bayes risk In estimation theory and decision theory, a Bayes estimator or a Bayes action is an estimator or decision rule that minimizes the posterior expected value of a loss function (i.e., the posterior expected loss). Equivalently, it maximizes the pos ...

(or an approximation thereof): Instead of taking the source sentence with maximal probability, we try to take the sentence that minimizes the expectancy of a given loss function with regards to all possible transcriptions (i.e., we take the sentence that minimizes the average distance to other possible sentences weighted by their estimated probability). The loss function is usually the Levenshtein distance, though it can be different distances for specific tasks; the set of possible transcriptions is, of course, pruned to maintain tractability. Efficient algorithms have been devised to re score lattices represented as weighted

finite state transducers A finite-state transducer (FST) is a finite-state machine with two memory ''tapes'', following the terminology for Turing machines: an input tape and an output tape. This contrasts with an ordinary finite-state automaton, which has a single tape. ...

with edit distances represented themselves as a finite state transducer verifying certain assumptions.

Dynamic time warping (DTW)-based speech recognition

Dynamic time warping is an approach that was historically used for speech recognition but has now largely been displaced by the more successful HMM-based approach. Dynamic time warping is an algorithm for measuring similarity between two sequences that may vary in time or speed. For instance, similarities in walking patterns would be detected, even if in one video the person was walking slowly and if in another he or she were walking more quickly, or even if there were accelerations and deceleration during the course of one observation. DTW has been applied to video, audio, and graphics – indeed, any data that can be turned into a linear representation can be analyzed with DTW. A well-known application has been automatic speech recognition, to cope with different speaking speeds. In general, it is a method that allows a computer to find an optimal match between two given sequences (e.g., time series) with certain restrictions. That is, the sequences are "warped" non-linearly to match each other. This sequence alignment method is often used in the context of hidden Markov models.

Neural networks

Neural networks emerged as an attractive acoustic modeling approach in ASR in the late 1980s. Since then, neural networks have been used in many aspects of speech recognition such as phoneme classification, phoneme classification through multi-objective evolutionary algorithms, isolated word recognition,

audiovisual speech recognition Audio visual speech recognition (AVSR) is a technique that uses image processing capabilities in lip reading to aid speech recognition systems in recognizing undeterministic Phone (phonetics), phones or giving preponderance among near probability de ...

, audiovisual speaker recognition and speaker adaptation.

Neural networks A neural network is a network or circuit of biological neurons, or, in a modern sense, an artificial neural network, composed of artificial neurons or nodes. Thus, a neural network is either a biological neural network, made up of biological ...

make fewer explicit assumptions about feature statistical properties than HMMs and have several qualities making them attractive recognition models for speech recognition. When used to estimate the probabilities of a speech feature segment, neural networks allow discriminative training in a natural and efficient manner. However, in spite of their effectiveness in classifying short-time units such as individual phonemes and isolated words, early neural networks were rarely successful for continuous recognition tasks because of their limited ability to model temporal dependencies. One approach to this limitation was to use neural networks as a pre-processing, feature transformation or dimensionality reduction, step prior to HMM based recognition. However, more recently, LSTM and related recurrent neural networks (RNNs) and Time Delay Neural Networks(TDNN's) have demonstrated improved performance in this area.

Deep feedforward and recurrent neural networks

Deep Neural Networks and Denoising Autoencoders are also under investigation. A deep feedforward neural network (DNN) is an

artificial neural network Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected unit ...

with multiple hidden layers of units between the input and output layers. Similar to shallow neural networks, DNNs can model complex non-linear relationships. DNN architectures generate compositional models, where extra layers enable composition of features from lower layers, giving a huge learning capacity and thus the potential of modeling complex patterns of speech data. A success of DNNs in large vocabulary speech recognition occurred in 2010 by industrial researchers, in collaboration with academic researchers, where large output layers of the DNN based on context dependent HMM states constructed by decision trees were adopted. Deng L., Li, J., Huang, J., Yao, K., Yu, D., Seide, F. et al
Recent Advances in Deep Learning for Speech Research at Microsoft
ICASSP, 2013. See comprehensive reviews of this development and of the state of the art as of October 2014 in the recent Springer book from Microsoft Research. See also the related background of automatic speech recognition and the impact of various machine learning paradigms, notably including

, in recent overview articles. One fundamental principle of

is to do away with hand-crafted

feature engineering Feature engineering or feature extraction or feature discovery is the process of using domain knowledge to extract features (characteristics, properties, attributes) from raw data. The motivation is to use these extra features to improve the qual ...

and to use raw features. This principle was first explored successfully in the architecture of deep autoencoder on the "raw" spectrogram or linear filter-bank features,L. Deng, M. Seltzer, D. Yu, A. Acero, A. Mohamed, and G. Hinton (2010
Binary Coding of Speech Spectrograms Using a Deep Auto-encoder
Interspeech. showing its superiority over the Mel-Cepstral features which contain a few stages of fixed transformation from spectrograms. The true "raw" features of speech, waveforms, have more recently been shown to produce excellent larger-scale speech recognition results.

End-to-end automatic speech recognition

Since 2014, there has been much research interest in "end-to-end" ASR. Traditional phonetic-based (i.e., all HMM-based model) approaches required separate components and training for the pronunciation, acoustic, and

. End-to-end models jointly learn all the components of the speech recognizer. This is valuable since it simplifies the training process and deployment process. For example, a n-gram language model is required for all HMM-based systems, and a typical n-gram language model often takes several gigabytes in memory making them impractical to deploy on mobile devices. Consequently, modern commercial ASR systems from

and

() are deployed on the cloud and require a network connection as opposed to the device locally. The first attempt at end-to-end ASR was with

Connectionist Temporal Classification Connectionist temporal classification (CTC) is a type of neural network output and associated scoring function, for training recurrent neural networks (RNNs) such as LSTM networks to tackle sequence problems where the timing is variable. It can b ...

(CTC)-based systems introduced by Alex Graves of

Google DeepMind DeepMind Technologies is a British artificial intelligence subsidiary of Alphabet Inc. and research laboratory founded in 2010. DeepMind was acquired by Google in 2014 and became a wholly owned subsidiary of Alphabet Inc, after Google's restru ...

and Navdeep Jaitly of the

University of Toronto The University of Toronto (UToronto or U of T) is a public research university in Toronto, Ontario, Canada, located on the grounds that surround Queen's Park. It was founded by royal charter in 1827 as King's College, the first institution ...

in 2014. The model consisted of recurrent neural networks and a CTC layer. Jointly, the RNN-CTC model learns the pronunciation and acoustic model together, however it is incapable of learning the language due to conditional independence assumptions similar to a HMM. Consequently, CTC models can directly learn to map speech acoustics to English characters, but the models make many common spelling mistakes and must rely on a separate language model to clean up the transcripts. Later, Baidu expanded on the work with extremely large datasets and demonstrated some commercial success in Chinese Mandarin and English. In 2016,

University of Oxford , mottoeng = The Lord is my light , established = , endowment = £6.1 billion (including colleges) (2019) , budget = £2.145 billion (2019–20) , chancellor ...

presented

LipNet LipNet is a deep neural network for visual speech recognition. It was created by Yannis Assael, Brendan Shillingford, Shimon Whiteson and Nando de Freitas, researchers from the University of Oxford. The technique, outlined in a paper in November ...

, the first end-to-end sentence-level lipreading model, using spatiotemporal convolutions coupled with an RNN-CTC architecture, surpassing human-level performance in a restricted grammar dataset. A large-scale CNN-RNN-CTC architecture was presented in 2018 by

achieving 6 times better performance than human experts. An alternative approach to CTC-based models are attention-based models. Attention-based ASR models were introduced simultaneously by Chan et al. of

Carnegie Mellon University Carnegie Mellon University (CMU) is a private research university in Pittsburgh, Pennsylvania. One of its predecessors was established in 1900 by Andrew Carnegie as the Carnegie Technical Schools; it became the Carnegie Institute of Technology ...

and Google Brain and Bahdanau et al. of the University of Montreal in 2016. The model named "Listen, Attend and Spell" (LAS), literally "listens" to the acoustic signal, pays "attention" to different parts of the signal and "spells" out the transcript one character at a time. Unlike CTC-based models, attention-based models do not have conditional-independence assumptions and can learn all the components of a speech recognizer including the pronunciation, acoustic and language model directly. This means, during deployment, there is no need to carry around a language model making it very practical for applications with limited memory. By the end of 2016, the attention-based models have seen considerable success including outperforming the CTC models (with or without an external language model). Various extensions have been proposed since the original LAS model. Latent Sequence Decompositions (LSD) was proposed by

, MIT and Google Brain to directly emit sub-word units which are more natural than English characters;

University of Oxford , mottoeng = The Lord is my light , established = , endowment = £6.1 billion (including colleges) (2019) , budget = £2.145 billion (2019–20) , chancellor ...

and

extended LAS to "Watch, Listen, Attend and Spell" (WLAS) to handle lip reading surpassing human-level performance.

Applications

In-car systems

Typically a manual control input, for example by means of a finger control on the steering-wheel, enables the speech recognition system and this is signaled to the driver by an audio prompt. Following the audio prompt, the system has a "listening window" during which it may accept a speech input for recognition. Simple voice commands may be used to initiate phone calls, select radio stations or play music from a compatible smartphone, MP3 player or music-loaded flash drive. Voice recognition capabilities vary between car make and model. Some of the most recent car models offer natural-language speech recognition in place of a fixed set of commands, allowing the driver to use full sentences and common phrases. With such systems there is, therefore, no need for the user to memorize a set of fixed command words.

Health care

Medical documentation

In the

health care Health care or healthcare is the improvement of health via the prevention, diagnosis, treatment, amelioration or cure of disease, illness, injury, and other physical and mental impairments in people. Health care is delivered by health profe ...

sector, speech recognition can be implemented in front-end or back-end of the medical documentation process. Front-end speech recognition is where the provider dictates into a speech-recognition engine, the recognized words are displayed as they are spoken, and the dictator is responsible for editing and signing off on the document. Back-end or deferred speech recognition is where the provider dictates into a

digital dictation A dictation machine is a sound recording device most commonly used to record speech for playback or to be typed into print. It includes digital voice recorders and tape recorder. The name "Dictaphone" is a trademark of the company of the same n ...

system, the voice is routed through a speech-recognition machine and the recognized draft document is routed along with the original voice file to the editor, where the draft is edited and report finalized. Deferred speech recognition is widely used in the industry currently. One of the major issues relating to the use of speech recognition in healthcare is that the American Recovery and Reinvestment Act of 2009 ( ARRA) provides for substantial financial benefits to physicians who utilize an EMR according to "Meaningful Use" standards. These standards require that a substantial amount of data be maintained by the EMR (now more commonly referred to as an

Electronic Health Record An electronic health record (EHR) is the systematized collection of patient and population electronically stored health information in a digital format. These records can be shared across different health care settings. Records are shared throu ...

or EHR). The use of speech recognition is more naturally suited to the generation of narrative text, as part of a radiology/pathology interpretation, progress note or discharge summary: the ergonomic gains of using speech recognition to enter structured discrete data (e.g., numeric values or codes from a list or a

controlled vocabulary Control may refer to: Basic meanings Economics and business * Control (management), an element of management * Control, an element of management accounting * Comptroller (or controller), a senior financial officer in an organization * Controlling ...

) are relatively minimal for people who are sighted and who can operate a keyboard and mouse. A more significant issue is that most EHRs have not been expressly tailored to take advantage of voice-recognition capabilities. A large part of the clinician's interaction with the EHR involves navigation through the user interface using menus, and tab/button clicks, and is heavily dependent on keyboard and mouse: voice-based navigation provides only modest ergonomic benefits. By contrast, many highly customized systems for radiology or pathology dictation implement voice "macros", where the use of certain phrases – e.g., "normal report", will automatically fill in a large number of default values and/or generate boilerplate, which will vary with the type of the exam – e.g., a chest X-ray vs. a gastrointestinal contrast series for a radiology system.

Therapeutic use

Prolonged use of speech recognition software in conjunction with

s has shown benefits to short-term-memory restrengthening in

brain AVM A cerebral arteriovenous malformation (cerebral AVM, CAVM, cAVM) is an abnormal connection between the arteries and veins in the brain—specifically, an arteriovenous malformation in the cerebrum. Signs and symptoms The most frequently observed ...

patients who have been treated with resection. Further research needs to be conducted to determine cognitive benefits for individuals whose AVMs have been treated using radiologic techniques.

Military

High-performance fighter aircraft

Substantial efforts have been devoted in the last decade to the test and evaluation of speech recognition in

fighter aircraft Fighter aircraft are fixed-wing military aircraft designed primarily for air-to-air combat. In military conflict, the role of fighter aircraft is to establish air superiority of the battlespace. Domination of the airspace above a battlefield ...

. Of particular note have been the US program in speech recognition for the Advanced Fighter Technology Integration (AFTI)/

F-16 The General Dynamics F-16 Fighting Falcon is a single-engine Multirole combat aircraft, multirole fighter aircraft originally developed by General Dynamics for the United States Air Force (USAF). Designed as an air superiority day fighter, it ...

aircraft (

F-16 VISTA The General Dynamics X-62 VISTA ("Variable Stability In-flight Simulator Test Aircraft") is an experimental aircraft, derived from the F-16D Fighting Falcon, which was modified as a joint venture between General Dynamics and Calspan for use by t ...

), the program in France for

Mirage A mirage is a naturally-occurring optical phenomenon in which light rays bend via refraction to produce a displaced image of distant objects or the sky. The word comes to English via the French ''(se) mirer'', from the Latin ''mirari'', meanin ...

aircraft, and other programs in the UK dealing with a variety of aircraft platforms. In these programs, speech recognizers have been operated successfully in fighter aircraft, with applications including setting radio frequencies, commanding an autopilot system, setting steer-point coordinates and weapons release parameters, and controlling flight display. Working with Swedish pilots flying in the

JAS-39 The Saab JAS 39 Gripen (; English: '' griffin'') is a light single-engine multirole fighter aircraft manufactured by the Swedish aerospace and defense company Saab AB. The Gripen has a delta wing and canard configuration with relaxed stab ...

Gripen cockpit, Englund (2004) found recognition deteriorated with increasing g-loads. The report also concluded that adaptation greatly improved the results in all cases and that the introduction of models for breathing was shown to improve recognition scores significantly. Contrary to what might have been expected, no effects of the broken English of the speakers were found. It was evident that spontaneous speech caused problems for the recognizer, as might have been expected. A restricted vocabulary, and above all, a proper syntax, could thus be expected to improve recognition accuracy substantially. The Eurofighter Typhoon, currently in service with the UK RAF, employs a speaker-dependent system, requiring each pilot to create a template. The system is not used for any safety-critical or weapon-critical tasks, such as weapon release or lowering of the undercarriage, but is used for a wide range of other cockpit functions. Voice commands are confirmed by visual and/or aural feedback. The system is seen as a major design feature in the reduction of pilot

workload The term workload can refer to a number of different yet related entities. An amount of labor Workload is the amount of work an individual has to do.Jex, S. M. (1998). Stress and job performance: Theory, research, and implications for managerial ...

, and even allows the pilot to assign targets to his aircraft with two simple voice commands or to any of his wingmen with only five commands. Speaker-independent systems are also being developed and are under test for the

F35 Lightning II The Lockheed Martin F-35 Lightning II is an American family of single-seat, single-engine, all-weather stealth multirole combat aircraft that is intended to perform both air superiority and strike missions. It is also able to provide ele ...

(JSF) and the

Alenia Aermacchi M-346 Master The Aermacchi M-346 Master is a family of military twin-engine transonic advanced jet trainers and light combat aircraft. Originally co-developed with Yakovlev as the Yak/AEM-130, the partnership was dissolved in 2000 and then Alenia Aermacchi ...

lead-in fighter trainer. These systems have produced word accuracy scores in excess of 98%.

Helicopters

The problems of achieving high recognition accuracy under stress and noise are particularly relevant in the

helicopter A helicopter is a type of rotorcraft in which lift and thrust are supplied by horizontally spinning rotors. This allows the helicopter to take off and land vertically, to hover, and to fly forward, backward and laterally. These attributes ...

environment as well as in the jet fighter environment. The acoustic noise problem is actually more severe in the helicopter environment, not only because of the high noise levels but also because the helicopter pilot, in general, does not wear a facemask, which would reduce acoustic noise in the

microphone A microphone, colloquially called a mic or mike (), is a transducer that converts sound into an electrical signal. Microphones are used in many applications such as telephones, hearing aids, public address systems for concert halls and public ...

. Substantial test and evaluation programs have been carried out in the past decade in speech recognition systems applications in helicopters, notably by the U.S. Army Avionics Research and Development Activity (AVRADA) and by the Royal Aerospace Establishment ( RAE) in the UK. Work in France has included speech recognition in the

Puma helicopter Puma or PUMA may refer to: Animals * ''Puma'' (genus), a genus in the family Felidae ** Puma (species) or cougar, a large cat Businesses and organisations * Puma (brand), a multinational shoe and sportswear company * Puma Energy, a mid- and d ...

. There has also been much useful work in

Canada Canada is a country in North America. Its ten provinces and three territories extend from the Atlantic Ocean to the Pacific Ocean and northward into the Arctic Ocean, covering over , making it the world's second-largest country by tot ...

. Results have been encouraging, and voice applications have included: control of communication radios, setting of

navigation Navigation is a field of study that focuses on the process of monitoring and controlling the movement of a craft or vehicle from one place to another.Bowditch, 2003:799. The field of navigation includes four general categories: land navigation, ...

systems, and control of an automated target handover system. As in fighter applications, the overriding issue for voice in helicopters is the impact on pilot effectiveness. Encouraging results are reported for the AVRADA tests, although these represent only a feasibility demonstration in a test environment. Much remains to be done both in speech recognition and in overall speech technology in order to consistently achieve performance improvements in operational settings.

Training air traffic controllers

Training for air traffic controllers (ATC) represents an excellent application for speech recognition systems. Many ATC training systems currently require a person to act as a "pseudo-pilot", engaging in a voice dialog with the trainee controller, which simulates the dialog that the controller would have to conduct with pilots in a real ATC situation. Speech recognition and

synthesis Synthesis or synthesize may refer to: Science Chemistry and biochemistry *Chemical synthesis, the execution of chemical reactions to form a more complex molecule from chemical precursors ** Organic synthesis, the chemical synthesis of organ ...

techniques offer the potential to eliminate the need for a person to act as a pseudo-pilot, thus reducing training and support personnel. In theory, Air controller tasks are also characterized by highly structured speech as the primary output of the controller, hence reducing the difficulty of the speech recognition task should be possible. In practice, this is rarely the case. The FAA document 7110.65 details the phrases that should be used by air traffic controllers. While this document gives less than 150 examples of such phrases, the number of phrases supported by one of the simulation vendors speech recognition systems is in excess of 500,000. The USAF, USMC, US Army, US Navy, and FAA as well as a number of international ATC training organizations such as the Royal Australian Air Force and Civil Aviation Authorities in Italy, Brazil, and Canada are currently using ATC simulators with speech recognition from a number of different vendors.

Telephony and other domains

ASR is now commonplace in the field of

telephony Telephony ( ) is the field of technology involving the development, application, and deployment of telecommunication services for the purpose of electronic transmission of voice, fax, or data, between distant parties. The history of telephony is i ...

and is becoming more widespread in the field of

computer gaming A personal computer game, also known as a PC game or computer game, is a type of video game played on a personal computer (PC) rather than a video game console or arcade cabinet, arcade machine. Its defining characteristics include: more divers ...

and simulation. In telephony systems, ASR is now being predominantly used in contact centers by integrating it with

IVR Interactive voice response (IVR) is a technology that allows telephone users to interact with a computer-operated telephone system through the use of voice and DTMF tones input with a keypad. In telecommunications, IVR allows customers to interact ...

systems. Despite the high level of integration with word processing in general personal computing, in the field of document production, ASR has not seen the expected increases in use. The improvement of mobile processor speeds has made speech recognition practical in

smartphone A smartphone is a portable computer device that combines mobile telephone and computing functions into one unit. They are distinguished from feature phones by their stronger hardware capabilities and extensive mobile operating systems, whic ...

s. Speech is used mostly as a part of a user interface, for creating predefined or custom speech commands.

Usage in education and daily life

For

language learning Language acquisition is the process by which humans acquire the capacity to perceive and comprehend language (in other words, gain the ability to be aware of language and to understand it), as well as to produce and use words and sentences to ...

, speech recognition can be useful for learning a

second language A person's second language, or L2, is a language that is not the native language (first language or L1) of the speaker, but is learned later. A second language may be a neighbouring language, another language of the speaker's home country, or a fo ...

. It can teach proper pronunciation, in addition to helping a person develop fluency with their speaking skills. Students who are blind (see

Blindness and education The subject of blindness and education has included evolving approaches and public perceptions of how best to address the special needs of blind students. The practice of institutionalizing the blind in asylums has a history extending back over a ...

) or have very low vision can benefit from using the technology to convey words and then hear the computer recite them, as well as use a computer by commanding with their voice, instead of having to look at the screen and keyboard. Students who are physically disabled , have a

Repetitive strain injury A repetitive strain injury (RSI) is an injury to part of the musculoskeletal or nervous system caused by repetitive use, vibrations, compression or long periods in a fixed position. Other common names include repetitive stress disorders, cumula ...

/other injuries to the upper extremities can be relieved from having to worry about handwriting, typing, or working with scribe on school assignments by using speech-to-text programs. They can also utilize speech recognition technology to enjoy searching the Internet or using a computer at home without having to physically operate a mouse and keyboard. Speech recognition can allow students with learning disabilities to become better writers. By saying the words aloud, they can increase the fluidity of their writing, and be alleviated of concerns regarding spelling, punctuation, and other mechanics of writing. Also, see

Learning disability Learning disability, learning disorder, or learning difficulty (British English) is a condition in the brain that causes difficulties comprehending or processing information and can be caused by several different factors. Given the "difficult ...

. The use of voice recognition software, in conjunction with a digital audio recorder and a personal computer running word-processing software has proven to be positive for restoring damaged short-term memory capacity, in stroke and craniotomy individuals.

People with disabilities

People with disabilities can benefit from speech recognition programs. For individuals that are Deaf or Hard of Hearing, speech recognition software is used to automatically generate a closed-captioning of conversations such as discussions in conference rooms, classroom lectures, and/or religious services. Speech recognition is also very useful for people who have difficulty using their hands, ranging from mild repetitive stress injuries to involve disabilities that preclude using conventional computer input devices. In fact, people who used the keyboard a lot and developed RSI became an urgent early market for speech recognition. Speech recognition is used in deaf

, such as voicemail to text,

relay services A telecommunications relay service, also known as TRS, relay service, or IP-relay, or Web-based relay service, is an operator service that allows people who are deaf, hard of hearing, deafblind, or have a speech disorder to place calls to sta ...

, and

captioned telephone A telecommunications relay service, also known as TRS, relay service, or IP-relay, or Web-based relay service, is an operator service that allows people who are deaf, hard of hearing, deafblind, or have a speech disorder to place calls to stan ...

. Individuals with learning disabilities who have problems with thought-to-paper communication (essentially they think of an idea but it is processed incorrectly causing it to end up differently on paper) can possibly benefit from the software but the technology is not bug proof. Also the whole idea of speak to text can be hard for intellectually disabled person's due to the fact that it is rare that anyone tries to learn the technology to teach the person with the disability. This type of technology can help those with dyslexia but other disabilities are still in question. The effectiveness of the product is the problem that is hindering it from being effective. Although a kid may be able to say a word depending on how clear they say it the technology may think they are saying another word and input the wrong one. Giving them more work to fix, causing them to have to take more time with fixing the wrong word.

Further applications

Aerospace Aerospace is a term used to collectively refer to the atmosphere and outer space. Aerospace activity is very diverse, with a multitude of commercial, industrial and military applications. Aerospace engineering consists of aeronautics and astrona ...

(e.g.

space exploration Space exploration is the use of astronomy and space technology to explore outer space. While the exploration of space is carried out mainly by astronomers with telescopes, its physical exploration though is conducted both by robotic spacec ...

spacecraft A spacecraft is a vehicle or machine designed to fly in outer space. A type of artificial satellite, spacecraft are used for a variety of purposes, including communications, Earth observation, meteorology, navigation, space colonization, p ...

, etc.) NASA's Mars Polar Lander used speech recognition technology from

Sensory, Inc. Sensory, Inc. is an American company which develops software AI technologies for speech, sound and vision . It is based in Santa Clara County, California, Santa Clara, California. Sensory's technologies have shipped in billions of products includ ...

in the Mars Microphone on the Lander *Automatic

subtitling Subtitles and captions are lines of dialogue or other text displayed at the bottom of the screen in films, television programs, video games or other visual media. They can be transcriptions of the screenplay, translations of it, or informati ...

with speech recognition *Automatic emotion recognition *Automatic

shot Shot may refer to: Arts, entertainment, and media * ''Shot'' (album), by The Jesus Lizard *''Shot, Illusion, New God'', an EP by Gruntruck *''Shot Rev 2.0'', a video album by The Sisters of Mercy * "Shot" (song), by The Rasmus * ''Shot'' (2017 fi ...

listing in audiovisual production *

Automatic translation Machine translation, sometimes referred to by the abbreviation MT (not to be confused with computer-aided translation, machine-aided human translation or interactive translation), is a sub-field of computational linguistics that investigates t ...

Court reporting A court reporter, court stenographer, or shorthand reporter is a person whose occupation is to capture the live testimony in proceedings using a stenographic machine, thereby transforming the proceedings into an official certified transcript b ...

(Real time Speech Writing) *

eDiscovery Electronic discovery (also ediscovery or e-discovery) refers to discovery in legal proceedings such as litigation, government investigations, or Freedom of Information Act requests, where the information sought is in electronic format (often refe ...

(Legal discovery) *

Hands-free computing Hands-free computing is any computer configuration where a user can interface without the use of their hands, an otherwise common requirement of human interface devices such as the mouse and keyboard. Hands-free computing is important because it is ...

: Speech recognition computer

user interface In the industrial design field of human–computer interaction, a user interface (UI) is the space where interactions between humans and machines occur. The goal of this interaction is to allow effective operation and control of the machine f ...

Home automation Home automation or domotics is building automation for a home, called a smart home or smart house. A home automation system will monitor and/or control home attributes such as lighting, climate, entertainment systems, and appliances. It m ...

Interactive voice response Interactive voice response (IVR) is a technology that allows telephone users to interact with a computer-operated telephone system through the use of voice and DTMF tones input with a keypad. In telecommunications, IVR allows customers to interact ...

Mobile telephony Mobile telephony is the provision of telephone services to phones which may move around freely rather than stay fixed in one location. Telephony is supposed to specifically point to a voice-only service or connection, though sometimes the li ...

, including mobile email * Multimodal interaction *

Pronunciation Pronunciation is the way in which a word or a language is spoken. This may refer to generally agreed-upon sequences of sounds used in speaking a given word or language in a specific dialect ("correct pronunciation") or simply the way a particular ...

evaluation in computer-aided language learning applications *Real Time

Captioning Closed captioning (CC) and subtitling are both processes of displaying text on a television, video screen, or other visual display to provide additional or interpretive information. Both are typically used as a transcription of the audio port ...

Robotics Robotics is an interdisciplinary branch of computer science and engineering. Robotics involves design, construction, operation, and use of robots. The goal of robotics is to design machines that can help and assist humans. Robotics integrat ...

*Security, including usage with other biometric scanners for multi-factor authentication *Speech to text (transcription of speech into text, real time video

captioning Closed captioning (CC) and subtitling are both processes of displaying text on a television, video screen, or other visual display to provide additional or interpretive information. Both are typically used as a transcription of the audio port ...

, Court reporting ) *

Telematics Telematics is an interdisciplinary field encompassing telecommunications, vehicular technologies ( road transport, road safety, etc.), electrical engineering (sensors, instrumentation, wireless communications, etc.), and computer science (multimedi ...

(e.g. vehicle Navigation Systems) *

Transcription Transcription refers to the process of converting sounds (voice, music etc.) into letters or musical notes, or producing a copy of something in another medium, including: Genetics * Transcription (biology), the copying of DNA into RNA, the fir ...

(digital speech-to-text) *

Video games Video games, also known as computer games, are electronic games that involves interaction with a user interface or input device such as a joystick, game controller, controller, computer keyboard, keyboard, or motion sensing device to gener ...

, with ''

Tom Clancy's EndWar ''Tom Clancy's EndWar'' is a strategy video game available on Microsoft Windows and all seventh-generation platforms except the Wii, with the timing and flow of gameplay differing across platforms. The console and PC version is a real-time tac ...

'' and '' Lifeline'' as working examples *

Virtual assistant An intelligent virtual assistant (IVA) or intelligent personal assistant (IPA) is a software agent that can perform tasks or services for an individual based on commands or questions. The term "chatbot" is sometimes used to refer to virtual ...

(e.g. Apple's Siri)

Performance

The performance of speech recognition systems is usually evaluated in terms of accuracy and speed. Accuracy is usually rated with word error rate (WER), whereas speed is measured with the

real time factor Real may refer to: Currencies * Brazilian real (R$) * Central American Republic real * Mexican real * Portuguese real * Spanish real * Spanish colonial real Music Albums * ''Real'' (L'Arc-en-Ciel album) (2000) * ''Real'' (Bright album) (2010) ...

. Other measures of accuracy include

Single Word Error Rate Single may refer to: Arts, entertainment, and media * Single (music), a song release Songs * "Single" (Natasha Bedingfield song), 2004 * "Single" (New Kids on the Block and Ne-Yo song), 2008 * "Single" (William Wei song), 2016 * "Single", by ...

(SWER) and

Command Success Rate Command may refer to: Computing * Command (computing), a statement in a computer language * COMMAND.COM, the default operating system shell and command-line interpreter for DOS * Command key, a modifier key on Apple Macintosh computer keyboards * ...

(CSR). Speech recognition by machine is a very complex problem, however. Vocalizations vary in terms of accent, pronunciation, articulation, roughness, nasality, pitch, volume, and speed. Speech is distorted by a background noise and echoes, electrical characteristics. Accuracy of speech recognition may vary with the following: * Vocabulary size and confusability * Speaker dependence versus independence * Isolated, discontinuous or continuous speech * Task and language constraints * Read versus spontaneous speech * Adverse conditions

Accuracy

As mentioned earlier in this article, the accuracy of speech recognition may vary depending on the following factors: * Error rates increase as the vocabulary size grows: ::e.g. the 10 digits "zero" to "nine" can be recognized essentially perfectly, but vocabulary sizes of 200, 5000 or 100000 may have error rates of 3%, 7%, or 45% respectively. * Vocabulary is hard to recognize if it contains confusing words: ::e.g. the 26 letters of the English alphabet are difficult to discriminate because they are confusing words (most notoriously, the E-set: "B, C, D, E, G, P, T, V, Z — when "Z" is pronounced "zee" rather than "zed" depending on the English region); an 8% error rate is considered good for this vocabulary. * Speaker dependence vs. independence: :: A speaker-dependent system is intended for use by a single speaker. :: A speaker-independent system is intended for use by any speaker (more difficult). * Isolated, Discontinuous or continuous speech :: With isolated speech, single words are used, therefore it becomes easier to recognize the speech. With discontinuous speech full sentences separated by silence are used, therefore it becomes easier to recognize the speech as well as with isolated speech.
With continuous speech naturally spoken sentences are used, therefore it becomes harder to recognize the speech, different from both isolated and discontinuous speech. * Task and language constraints **e.g. Querying application may dismiss the hypothesis "The apple is red." **e.g. Constraints may be semantic; rejecting "The apple is angry." **e.g. Syntactic; rejecting "Red is apple the." Constraints are often represented by grammar. * Read vs. Spontaneous Speech – When a person reads it's usually in a context that has been previously prepared, but when a person uses spontaneous speech, it is difficult to recognize the speech because of the disfluencies (like "uh" and "um", false starts, incomplete sentences, stuttering, coughing, and laughter) and limited vocabulary. * Adverse conditions – Environmental noise (e.g. Noise in a car or a factory). Acoustical distortions (e.g. echoes, room acoustics) Speech recognition is a multi-leveled pattern recognition task. * Acoustical signals are structured into a hierarchy of units, e.g.

Phoneme In phonology and linguistics, a phoneme () is a unit of sound that can distinguish one word from another in a particular language. For example, in most dialects of English, with the notable exception of the West Midlands and the north-west o ...

s, Words, Phrases, and Sentences; * Each level provides additional constraints; e.g. Known word pronunciations or legal word sequences, which can compensate for errors or uncertainties at a lower level; * This hierarchy of constraints is exploited. By combining decisions probabilistically at all lower levels, and making more deterministic decisions only at the highest level, speech recognition by a machine is a process broken into several phases. Computationally, it is a problem in which a sound pattern has to be recognized or classified into a category that represents a meaning to a human. Every acoustic signal can be broken into smaller more basic sub-signals. As the more complex sound signal is broken into the smaller sub-sounds, different levels are created, where at the top level we have complex sounds, which are made of simpler sounds on the lower level, and going to lower levels, even more, we create more basic and shorter and simpler sounds. At the lowest level, where the sounds are the most fundamental, a machine would check for simple and more probabilistic rules of what sound should represent. Once these sounds are put together into more complex sounds on upper level, a new set of more deterministic rules should predict what the new complex sound should represent. The most upper level of a deterministic rule should figure out the meaning of complex expressions. In order to expand our knowledge about speech recognition, we need to take into consideration neural networks. There are four steps of neural network approaches: * Digitize the speech that we want to recognize For telephone speech the sampling rate is 8000 samples per second; * Compute features of spectral-domain of the speech (with Fourier transform); computed every 10 ms, with one 10 ms section called a frame; Analysis of four-step neural network approaches can be explained by further information. Sound is produced by air (or some other medium) vibration, which we register by ears, but machines by receivers. Basic sound creates a wave which has two descriptions:

amplitude The amplitude of a periodic variable is a measure of its change in a single period (such as time or spatial period). The amplitude of a non-periodic signal is its magnitude compared with a reference value. There are various definitions of amplit ...

(how strong is it), and

frequency Frequency is the number of occurrences of a repeating event per unit of time. It is also occasionally referred to as ''temporal frequency'' for clarity, and is distinct from ''angular frequency''. Frequency is measured in hertz (Hz) which is eq ...

(how often it vibrates per second). Accuracy can be computed with the help of word error rate (WER). Word error rate can be calculated by aligning the recognized word and referenced word using dynamic string alignment. The problem may occur while computing the word error rate due to the difference between the sequence lengths of the recognized word and referenced word. Let S be the number of substitutions, D be the number of deletions, I be the number of insertions, N be the number of word references. The formula to compute the word error rate(WER) is WER = (S+D+I)÷N While computing the word recognition rate (WRR) word error rate (WER) is used and the formula is WRR = 1- WER = (N-S-D-I)÷ N = (H-I)÷N Here H is the number of correctly recognized words. H= N-(S+D).

Security concerns

Speech recognition can become a means of attack, theft, or accidental operation. For example, activation words like "Alexa" spoken in an audio or video broadcast can cause devices in homes and offices to start listening for input inappropriately, or possibly take an unwanted action. Voice-controlled devices are also accessible to visitors to the building, or even those outside the building if they can be heard inside. Attackers may be able to gain access to personal information, like calendar, address book contents, private messages, and documents. They may also be able to impersonate the user to send messages or make online purchases. Two attacks have been demonstrated that use artificial sounds. One transmits ultrasound and attempt to send commands without nearby people noticing. The other adds small, inaudible distortions to other speech or music that are specially crafted to confuse the specific speech recognition system into recognizing music as speech, or to make what sounds like one command to a human sound like a different command to the system.

Further information

Conferences and journals

Popular speech recognition conferences held each year or two include SpeechTEK and SpeechTEK Europe,

, Interspeech/Eurospeech, and the IEEE ASRU. Conferences in the field of

natural language processing Natural language processing (NLP) is an interdisciplinary subfield of linguistics, computer science, and artificial intelligence concerned with the interactions between computers and human language, in particular how to program computers to pro ...

, such as ACL, NAACL, EMNLP, and HLT, are beginning to include papers on speech processing. Important journals include the

IEEE The Institute of Electrical and Electronics Engineers (IEEE) is a 501(c)(3) professional association for electronic engineering and electrical engineering (and associated disciplines) with its corporate office in New York City and its operation ...

Transactions on Speech and Audio Processing (later renamed

Transactions on Audio, Speech and Language Processing and since Sept 2014 renamed

/ACM Transactions on Audio, Speech and Language Processing—after merging with an ACM publication), Computer Speech and Language, and Speech Communication.

Books

Books like "Fundamentals of Speech Recognition" by

can be useful to acquire basic knowledge but may not be fully up to date (1993). Another good source can be "Statistical Methods for Speech Recognition" by

Frederick Jelinek Frederick Jelinek (18 November 1932 – 14 September 2010) was a Czech-American researcher in information theory, automatic speech recognition, and natural language processing. He is well known for his oft-quoted statement, "Every time I fire a ...

and "Spoken Language Processing (2001)" by

etc., "Computer Speech", by

Manfred R. Schroeder Manfred Robert Schroeder (12 July 1926 – 28 December 2009) was a German physicist, most known for his contributions to acoustics and computer graphics. He wrote three books and published over 150 articles in his field. Born in Ahlen, he stud ...

, second edition published in 2004, and "Speech Processing: A Dynamic and Optimization-Oriented Approach" published in 2003 by Li Deng and Doug O'Shaughnessey. The updated textbook ''Speech and Language Processing'' (2008) by Jurafsky and Martin presents the basics and the state of the art for ASR.

Speaker recognition Speaker recognition is the identification of a person from characteristics of voices. It is used to answer the question "Who is speaking?" The term voice recognition can refer to ''speaker recognition'' or speech recognition. Speaker verification ...

also uses the same features, most of the same front-end processing, and classification techniques as is done in speech recognition. A comprehensive textbook, "Fundamentals of Speaker Recognition" is an in depth source for up to date details on the theory and practice. A good insight into the techniques used in the best modern systems can be gained by paying attention to government sponsored evaluations such as those organised by

(the largest speech recognition-related project ongoing as of 2007 is the GALE project, which involves both speech recognition and translation components). A good and accessible introduction to speech recognition technology and its history is provided by the general audience book "The Voice in the Machine. Building Computers That Understand Speech" by

Roberto Pieraccini Roberto Pieraccini (born 15 November 1955 in Genoa, Italy) is an Italian and US electrical engineer working in the field of speech recognition, natural language understanding, and spoken dialog systems. He is currently Director of Engineering a ...

(2012). The most recent book on speech recognition is ''Automatic Speech Recognition: A Deep Learning Approach'' (Publisher: Springer) written by Microsoft researchers D. Yu and L. Deng and published near the end of 2014, with highly mathematically oriented technical detail on how deep learning methods are derived and implemented in modern speech recognition systems based on DNNs and related deep learning methods. A related book, published earlier in 2014, "Deep Learning: Methods and Applications" by L. Deng and D. Yu provides a less technical but more methodology-focused overview of DNN-based speech recognition during 2009–2014, placed within the more general context of deep learning applications including not only speech recognition but also image recognition, natural language processing, information retrieval, multimodal processing, and multitask learning.

Software

In terms of freely available resources,

's Sphinx toolkit is one place to start to both learn about speech recognition and to start experimenting. Another resource (free but copyrighted) is the HTK book (and the accompanying HTK toolkit). For more recent and state-of-the-art techniques,

Kaldi Kaldi or Khalid was a legendary Arab Ethiopian goatherd who discovered the coffee plant around 850 CE, according to popular legend, show some artwork depicting him, after which it entered the Islamic world and then the rest of the world. Story I ...

toolkit can be used. In 2017

Mozilla Mozilla (stylized as moz://a) is a free software community founded in 1998 by members of Netscape. The Mozilla community uses, develops, spreads and supports Mozilla products, thereby promoting exclusively free software and open standards, wi ...

launched the open source project called

Common Voice Common Voice is a crowdsourcing project started by Mozilla to create a free database for speech recognition software. The project is supported by volunteers who record sample sentences with a microphone and review recordings of other users. The t ...

to gather big database of voices that would help build free speech recognition project

DeepSpeech Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the mai ...

(available free at

GitHub GitHub, Inc. () is an Internet hosting service for software development and version control using Git. It provides the distributed version control of Git plus access control, bug tracking, software feature requests, task management, continuous ...

), using Google's open source platform TensorFlow. When Mozilla redirected funding away from the project in 2020, it was forked by its original developers as Coqui STT using the same open-source license. The commercial cloud based speech recognition APIs are broadly available. For more software resources, see

List of speech recognition software Speech recognition software is available for many computing platforms, operating systems, use models, and software licenses. Here is a listing of such, grouped in various useful ways. Acoustic models and speech corpus (compilation) The following l ...

References

External links

* Signer, Beat and Hoste, Lode
''SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry''
In Proceedings of ICMI 2013, 15th International Conference on Multimodal Interaction, Sydney, Australia, December 2013 * {{DEFAULTSORT:Speech Recognition Speech recognition, Automatic identification and data capture Computational linguistics User interface techniques History of human–computer interaction Computer accessibility Machine learning task