HOME

TheInfoList



OR:

A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the
logistic function A logistic function or logistic curve is a common S-shaped curve (sigmoid function, sigmoid curve) with equation f(x) = \frac, where For values of x in the domain of real numbers from -\infty to +\infty, the S-curve shown on the right is ...
shown in the first figure and defined by the formula: :S(x) = \frac = \frac=1-S(-x). Other standard sigmoid functions are given in the Examples section. In some fields, most notably in the context of
artificial neural network Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected units ...
s, the term "sigmoid function" is used as an alias for the logistic function. Special cases of the sigmoid function include the Gompertz curve (used in modeling systems that saturate at large values of x) and the ogee curve (used in the
spillway A spillway is a structure used to provide the controlled release of water downstream from a dam or levee, typically into the riverbed of the dammed river itself. In the United Kingdom, they may be known as overflow channels. Spillways ensure t ...
of some dams). Sigmoid functions have domain of all
real number In mathematics, a real number is a number that can be used to measurement, measure a ''continuous'' one-dimensional quantity such as a distance, time, duration or temperature. Here, ''continuous'' means that values can have arbitrarily small var ...
s, with return (response) value commonly
monotonically increasing In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
but could be decreasing. Sigmoid functions most often show a return value (y axis) in the range 0 to 1. Another commonly used range is from −1 to 1. A wide variety of sigmoid functions including the logistic and hyperbolic tangent functions have been used as the activation function of artificial neurons. Sigmoid curves are also common in statistics as
cumulative distribution function In probability theory and statistics, the cumulative distribution function (CDF) of a real-valued random variable X, or just distribution function of X, evaluated at x, is the probability that X will take a value less than or equal to x. Ev ...
s (which go from 0 to 1), such as the integrals of the logistic density, the
normal density In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real number, real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The param ...
, and Student's ''t'' probability density functions. The logistic sigmoid function is invertible, and its inverse is the
logit In statistics, the logit ( ) function is the quantile function associated with the standard logistic distribution. It has many uses in data analysis and machine learning, especially in data transformations. Mathematically, the logit is the i ...
function.


Definition

A sigmoid function is a
bounded Boundedness or bounded may refer to: Economics * Bounded rationality, the idea that human rationality in decision-making is bounded by the available information, the cognitive limitations, and the time available to make the decision * Bounded e ...
, differentiable, real function that is defined for all real input values and has a non-negative derivative at each point and exactly one
inflection point In differential calculus and differential geometry, an inflection point, point of inflection, flex, or inflection (British English: inflexion) is a point on a smooth plane curve at which the curvature changes sign. In particular, in the case ...
. A sigmoid "function" and a sigmoid "curve" refer to the same object.


Properties

In general, a sigmoid function is
monotonic In mathematics, a monotonic function (or monotone function) is a function between ordered sets that preserves or reverses the given order. This concept first arose in calculus, and was later generalized to the more abstract setting of ord ...
, and has a first
derivative In mathematics, the derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value). Derivatives are a fundamental tool of calculus. ...
which is bell shaped. Conversely, the
integral In mathematics, an integral assigns numbers to functions in a way that describes displacement, area, volume, and other concepts that arise by combining infinitesimal data. The process of finding integrals is called integration. Along with ...
of any continuous, non-negative, bell-shaped function (with one local maximum and no local minimum, unless degenerate) will be sigmoidal. Thus the
cumulative distribution function In probability theory and statistics, the cumulative distribution function (CDF) of a real-valued random variable X, or just distribution function of X, evaluated at x, is the probability that X will take a value less than or equal to x. Ev ...
s for many common
probability distribution In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomeno ...
s are sigmoidal. One such example is the
error function In mathematics, the error function (also called the Gauss error function), often denoted by , is a complex function of a complex variable defined as: :\operatorname z = \frac\int_0^z e^\,\mathrm dt. This integral is a special (non- elementa ...
, which is related to the cumulative distribution function of a
normal distribution In statistics, a normal distribution or Gaussian distribution is a type of continuous probability distribution for a real-valued random variable. The general form of its probability density function is : f(x) = \frac e^ The parameter \mu i ...
; another is the
arctan In mathematics, the inverse trigonometric functions (occasionally also called arcus functions, antitrigonometric functions or cyclometric functions) are the inverse functions of the trigonometric functions (with suitably restricted domains). S ...
function, which is related to the cumulative distribution function of a
Cauchy distribution The Cauchy distribution, named after Augustin Cauchy, is a continuous probability distribution. It is also known, especially among physicists, as the Lorentz distribution (after Hendrik Lorentz), Cauchy–Lorentz distribution, Lorentz(ian) fu ...
. A sigmoid function is constrained by a pair of
horizontal asymptote In analytic geometry, an asymptote () of a curve is a line such that the distance between the curve and the line approaches zero as one or both of the ''x'' or ''y'' coordinates tends to infinity. In projective geometry and related contexts, ...
s as x \rightarrow \pm \infty. A sigmoid function is
convex Convex or convexity may refer to: Science and technology * Convex lens, in optics Mathematics * Convex set, containing the whole line segment that joins points ** Convex polygon, a polygon which encloses a convex set of points ** Convex polytop ...
for values less than a particular point, and it is concave for values greater than that point: in many of the examples here, that point is 0.


Examples

*
Logistic function A logistic function or logistic curve is a common S-shaped curve (sigmoid function, sigmoid curve) with equation f(x) = \frac, where For values of x in the domain of real numbers from -\infty to +\infty, the S-curve shown on the right is ...
f(x) = \frac * Hyperbolic tangent (shifted and scaled version of the logistic function, above) f(x) = \tanh x = \frac *
Arctangent function In mathematics, the inverse trigonometric functions (occasionally also called arcus functions, antitrigonometric functions or cyclometric functions) are the inverse functions of the trigonometric functions (with suitably restricted domains). Spe ...
f(x) = \arctan x *
Gudermannian function In mathematics, the Gudermannian function relates a hyperbolic angle measure \psi to a circular angle measure \phi called the ''gudermannian'' of \psi and denoted \operatorname\psi. The Gudermannian function reveals a close relationship betwee ...
f(x) = \operatorname(x) = \int_0^x \frac = 2\arctan\left(\tanh\left(\frac\right)\right) *
Error function In mathematics, the error function (also called the Gauss error function), often denoted by , is a complex function of a complex variable defined as: :\operatorname z = \frac\int_0^z e^\,\mathrm dt. This integral is a special (non- elementa ...
f(x) = \operatorname(x) = \frac \int_0^x e^ \, dt * Generalised logistic function f(x) = \left(1 + e^ \right)^, \quad \alpha > 0 * Smoothstep function f(x) = \begin , & , x, \le 1 \\ \\ \sgn(x) & , x, \ge 1 \\ \end \quad N \in \mathbb \ge 1 * Some
algebraic function In mathematics, an algebraic function is a function that can be defined as the root of a polynomial equation. Quite often algebraic functions are algebraic expressions using a finite number of terms, involving only the algebraic operations additi ...
s, for example f(x) = \frac * and in a more general form f(x) = \frac * Up to shifts and scaling, many sigmoids are special cases of f(x) = \varphi(\varphi(x, \beta), \alpha) , where \varphi(x, \lambda) = \begin (1 - \lambda x)^ & \lambda \ne 0 \\e^ & \lambda = 0 \\ \end is the inverse of the negative
Box–Cox transformation In statistics, a power transform is a family of functions applied to create a monotonic transformation of data using power functions. It is a data transformation technique used to stabilize variance, make the data more normal distribution-like ...
, and \alpha < 1 and \beta < 1 are shape parameters. * Smooth Interpolation normalized to (-1,1) and n is the slope at zero: \beginf(x) &= \begin , n=2 & , x, < 1 \\ \\ \sgn(x) & , x, \ge 1 \\ \end \\ &= \begin , n=2 & , x, < 1 \\ \\ \sgn(x) & , x, \ge 1 \\ \end\end using the hyperbolic tangent mentioned above.


Applications

Many natural processes, such as those of complex system learning curves, exhibit a progression from small beginnings that accelerates and approaches a climax over time. When a specific mathematical model is lacking, a sigmoid function is often used. The van Genuchten–Gupta model is based on an inverted S-curve and applied to the response of crop yield to
soil salinity Soil salinity is the salt content in the soil; the process of increasing the salt content is known as salinization. Salts occur naturally within soils and water. Salination can be caused by natural processes such as mineral weathering or by the ...
. Examples of the application of the logistic S-curve to the response of crop yield (wheat) to both the soil salinity and depth to
water table The water table is the upper surface of the zone of saturation. The zone of saturation is where the pores and fractures of the ground are saturated with water. It can also be simply explained as the depth below which the ground is saturated. Th ...
in the soil are shown in modeling crop response in agriculture. In
artificial neural network Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains. An ANN is based on a collection of connected units ...
s, sometimes non-smooth functions are used instead for efficiency; these are known as hard sigmoids. In
audio signal processing Audio signal processing is a subfield of signal processing that is concerned with the electronic manipulation of audio signals. Audio signals are electronic representations of sound waves— longitudinal waves which travel through air, consist ...
, sigmoid functions are used as
waveshaper In electronic music, waveshaping is a type of distortion synthesis in which complex spectra are produced from simple tones by altering the shape of the waveforms. Uses Waveshapers are used mainly by electronic musicians to achieve an extra ...
transfer function In engineering, a transfer function (also known as system function or network function) of a system, sub-system, or component is a mathematical function that theoretically models the system's output for each possible input. They are widely used ...
s to emulate the sound of analog circuitry clipping. In
biochemistry Biochemistry or biological chemistry is the study of chemical processes within and relating to living organisms. A sub-discipline of both chemistry and biology, biochemistry may be divided into three fields: structural biology, enzymology ...
and
pharmacology Pharmacology is a branch of medicine, biology and pharmaceutical sciences concerned with drug or medication action, where a drug may be defined as any artificial, natural, or endogenous (from within the body) molecule which exerts a biochemi ...
, the
Hill A hill is a landform that extends above the surrounding terrain. It often has a distinct summit. Terminology The distinction between a hill and a mountain is unclear and largely subjective, but a hill is universally considered to be not as ...
and Hill–Langmuir equations are sigmoid functions. In computer graphics and real-time rendering, some of the sigmoid functions are used to blend colors or geometry between two values, smoothly and without visible seams or discontinuities. Titration curves between strong acids and strong bases have a sigmoid shape due to the logarithmic nature of the
pH scale In chemistry, pH (), historically denoting "potential of hydrogen" (or "power of hydrogen"), is a scale used to specify the acidity or basicity of an aqueous solution. Acidic solutions (solutions with higher concentrations of ions) are mea ...
. The logistic function can be calculated efficiently by utilizing type III Unums.


See also

*
Heaviside step function The Heaviside step function, or the unit step function, usually denoted by or (but sometimes , or ), is a step function, named after Oliver Heaviside (1850–1925), the value of which is zero for negative arguments and one for positive argume ...
*
Logistic regression In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear function (calculus), linear combination of one or more independent var ...
*
Logit In statistics, the logit ( ) function is the quantile function associated with the standard logistic distribution. It has many uses in data analysis and machine learning, especially in data transformations. Mathematically, the logit is the i ...
*
Softplus function In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument: : f(x) = x^+ = \max(0, x), where ''x'' is the input to a neu ...
* Soboleva modified hyperbolic tangent * Softmax function * Swish function *
Weibull distribution In probability theory and statistics, the Weibull distribution is a continuous probability distribution. It is named after Swedish mathematician Waloddi Weibull, who described it in detail in 1951, although it was first identified by Maurice R ...
* Fermi–Dirac statistics


References


Further reading

* . (NB. In particular see "Chapter 4: Artificial Neural Networks" (in particular pp. 96–97) where Mitchell uses the word "logistic function" and the "sigmoid function" synonymously – this function he also calls the "squashing function" – and the sigmoid (aka logistic) function is used to compress the outputs of the "neurons" in multi-layer neural nets.) * (NB. Properties of the sigmoid, including how it can shift along axes and how its domain may be transformed.)


External links

* {{Differentiable computing Elementary special functions Artificial neural networks