openSMILE is

source-available Source-available software is software released through a source code distribution model that includes arrangements where the source can be viewed, and in some cases modified, but without necessarily meeting the criteria to be called open-source ...

software for automatic extraction of features from audio signals and for classification of speech and music signals. "SMILE" stands for "Speech & Music Interpretation by Large-space Extraction". The software is mainly applied in the area of automatic

emotion recognition Emotion recognition is the process of identifying human emotion. People vary widely in their accuracy at recognizing the emotions of others. Use of technology to help people with emotion recognition is a relatively nascent research area. Genera ...

and is widely used in the

affective computing Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. It is an interdisciplinary field spanning computer science, psychology, and cognitive science. While some ...

research community. The openSMILE project exists since 2008 and is maintained by the German company audEERING GmbH since 2013. openSMILE is provided free of charge for research purposes and personal use under a source-available license. For commercial use of the tool, the company audEERING offers custom license options.

Application Areas

openSMILE is used for academic research as well as for commercial applications in order to automatically analyze speech and music signals in real-time. In contrast to

automatic speech recognition Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the ...

which extracts the spoken content out of a speech signal, openSMILE is capable of recognizing the characteristics of a given speech or music segment. Examples for such characteristics encoded in human speech are a speaker's

emotion Emotions are mental states brought on by neurophysiology, neurophysiological changes, variously associated with thoughts, feelings, behavioral responses, and a degree of pleasure or suffering, displeasure. There is currently no scientific ...

, age, gender, and personality, as well as speaker states like depression,

intoxication Intoxication — or poisoning, especially by an alcoholic or narcotic substance — may refer to: * Substance intoxication: ** Alcohol intoxication ** LSD intoxication ** Toxidrome ** Tobacco intoxication ** Cannabis intoxication ** Cocaine i ...

, or vocal pathological disorders. The software further includes music classification technology for automatic music mood detection and recognition of chorus segments, key,

chords Chord may refer to: * Chord (music), an aggregate of musical pitches sounded simultaneously ** Guitar chord a chord played on a guitar, which has a particular tuning * Chord (geometry), a line segment joining two points on a curve * Chord ( ...

, tempo, meter, dance-style, and genre. The openSMILE toolkit serves as benchmark in manifold research competitions such as Interspeech ComParE, AVEC, MediaEval, and EmotiW.

History

The openSMILE project was started in 2008 by Florian Eyben, Martin Wöllmer, and Björn Schuller at the

Technical University of Munich The Technical University of Munich (TUM or TU Munich; german: Technische Universität München) is a public research university in Munich, Germany. It specializes in engineering, technology, medicine, and applied and natural sciences. Establis ...

within the

European Union The European Union (EU) is a supranational political and economic union of member states that are located primarily in Europe. The union has a total area of and an estimated total population of about 447million. The EU has often been de ...

research project SEMAINE. The goal of the SEMAINE project was to develop a virtual agent with emotional and

social intelligence Social intelligence is the capacity to know oneself and to know others. Social intelligence is learned and develops from experience with people and learning from success and failures in social settings. Social intelligence is the ability to underst ...

. In this system, openSMILE was applied for real-time analysis of speech and emotion. The final SEMAINE software release is based on openSMILE version 1.0.1. In 2009, the emotion recognition toolkit (openEAR) was published based on openSMILE. "EAR" stands for "Emotion and Affect Recognition". In 2010, openSMILE version 1.0.1 was published and was introduced and awarded at the

ACM Multimedia ACM Multimedia (ACM-MM) is the Association for Computing Machinery (ACM)'s annual conference on multimedia, sponsored by the SIGMM on multimedia in the ACM. SIGMM specializes in the field of multimedia computing, from underlying technologies to ...

Open-Source Software Challenge. Between 2011 and 2013, the technology of openSMILE was extended and improved by Florian Eyben and Felix Weninger in the context of their doctoral thesis at the

. The software was also applied for the project ASC-Inclusion, which was funded by the

. For this project, the software was extended by Erik Marchi in order to teach emotional expression to

autistic The autism spectrum, often referred to as just autism or in the context of a professional diagnosis autism spectrum disorder (ASD) or autism spectrum condition (ASC), is a neurodevelopmental condition (or conditions) characterized by difficulti ...

children, based on automatic emotion recognition and visualization. In 2013, the company audEERING acquired the rights to the code-base from the

and version 2.0 was published under a source-available research license. Until 2016, openSMILE was downloaded more than 50,000 times worldwide and has established itself as a standard toolkit for emotion recognition.

Awards

openSMILE was awarded in 2010 in the context of the

Open Source Competition. The software tool is applied in numerous scientific publications on automatic emotion recognition. openSMILE and its extension openEAR have been cited in more than 1000 scientific publications until today.

References

{{reflist

Weblinks

openSMILE website

openSMILE on GitHub

openSMILE documentation

Google Scholar page on openSMILE

Google Scholar page on openEAR

Article on startupvalley.com
Audio software Affective computing