Scale-space theory is a framework for
multi-scale signal
In signal processing, a signal is a function that conveys information about a phenomenon. Any quantity that can vary over space or time can be used as a signal to share messages between observers. The '' IEEE Transactions on Signal Processing' ...
representation developed by the
computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the hum ...
,
image processing
An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
and
signal processing
Signal processing is an electrical engineering subfield that focuses on analyzing, modifying and synthesizing ''signals'', such as audio signal processing, sound, image processing, images, and scientific measurements. Signal processing techniq ...
communities with complementary motivations from
physics
Physics is the natural science that studies matter, its fundamental constituents, its motion and behavior through space and time, and the related entities of energy and force. "Physical science is that department of knowledge which r ...
and
biological vision
Visual perception is the ability to interpret the surrounding environment through photopic vision (daytime vision), color vision, scotopic vision (night vision), and mesopic vision (twilight vision), using light in the visible spectrum reflect ...
. It is a formal theory for handling image structures at different
scales, by representing an image as a one-parameter family of smoothed images, the scale-space representation, parametrized by the size of the
smoothing
In statistics and image processing, to smooth a data set is to create an approximating function that attempts to capture important patterns in the data, while leaving out noise or other fine-scale structures/rapid phenomena. In smoothing, the dat ...
kernel
Kernel may refer to:
Computing
* Kernel (operating system), the central component of most operating systems
* Kernel (image processing), a matrix used for image convolution
* Compute kernel, in GPGPU programming
* Kernel method, in machine learn ...
used for suppressing fine-scale structures.
[Ijima, T. "Basic theory on normalization of pattern (in case of typical one-dimensional pattern)". Bull. Electrotech. Lab. 26, 368– 388, 1962. (in Japanese)] The parameter
in this family is referred to as the ''scale parameter'', with the interpretation that image structures of spatial size smaller than about
have largely been smoothed away in the scale-space level at scale
.
The main type of scale space is the ''linear (Gaussian) scale space'', which has wide applicability as well as the attractive property of being possible to derive from a small set of ''
scale-space axioms
In image processing and computer vision, a scale space framework can be used to represent an image as a family of gradually smoothed images. This framework is very general and a variety of scale space representations exist. A typical approach ...
''. The corresponding scale-space framework encompasses a theory for Gaussian derivative operators, which can be used as a basis for expressing a large class of visual operations for computerized systems that process visual information. This framework also allows visual operations to be made ''
scale invariant
In physics, mathematics and statistics, scale invariance is a feature of objects or laws that do not change if scales of length, energy, or other variables, are multiplied by a common factor, and thus represent a universality.
The technical ter ...
'', which is necessary for dealing with the size variations that may occur in image data, because real-world objects may be of different sizes and in addition the distance between the object and the camera may be unknown and may vary depending on the circumstances.
Definition
The notion of scale space applies to signals of arbitrary numbers of variables. The most common case in the literature applies to two-dimensional images, which is what is presented here. For a given image
, its linear (Gaussian) ''scale-space representation'' is a family of derived signals
defined by the
convolution
In mathematics (in particular, functional analysis), convolution is a operation (mathematics), mathematical operation on two function (mathematics), functions ( and ) that produces a third function (f*g) that expresses how the shape of one is ...
of
with the two-dimensional
Gaussian kernel
In mathematics, a Gaussian function, often simply referred to as a Gaussian, is a function of the base form
f(x) = \exp (-x^2)
and with parametric extension
f(x) = a \exp\left( -\frac \right)
for arbitrary real constants , and non-zero . It is ...
:
such that
:
where the semicolon in the argument of
implies that the convolution is performed only over the variables
, while the scale parameter
after the semicolon just indicates which scale level is being defined. This definition of
works for a continuum of scales
, but typically only a finite discrete set of levels in the scale-space representation would be actually considered.
The scale parameter
is the
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...
of the
Gaussian filter
In electronics and signal processing mainly in digital signal processing, a Gaussian filter is a filter whose impulse response is a Gaussian function (or an approximation to it, since a true Gaussian response would have infinite impulse respons ...
and as a limit for
the filter
becomes an
impulse function
In mathematics, the Dirac delta distribution ( distribution), also known as the unit impulse, is a generalized function or distribution over the real numbers, whose value is zero everywhere except at zero, and whose integral over the entire ...
such that
that is, the scale-space representation at scale level
is the image
itself. As
increases,
is the result of smoothing
with a larger and larger filter, thereby removing more and more of the details that the image contains. Since the standard deviation of the filter is
, details that are significantly smaller than this value are to a large extent removed from the image at scale parameter
, see the following figure and
for graphical illustrations.
Image:Scalespace0.png, Scale-space representation at scale , corresponding to the original image
Image:Scalespace1.png, Scale-space representation at scale
Image:Scalespace2.png, Scale-space representation at scale
Image:Scalespace3.png, Scale-space representation at scale
Image:Scalespace4.png, Scale-space representation at scale
Image:Scalespace5.png, Scale-space representation at scale
Why a Gaussian filter?
When faced with the task of generating a multi-scale representation one may ask: could any filter ''g'' of low-pass type and with a parameter ''t'' which determines its width be used to generate a scale space? The answer is no, as it is of crucial importance that the smoothing filter does not introduce new spurious structures at coarse scales that do not correspond to simplifications of corresponding structures at finer scales. In the scale-space literature, a number of different ways have been expressed to formulate this criterion in precise mathematical terms.
The conclusion from several different axiomatic derivations that have been presented is that the Gaussian scale space constitutes the ''canonical'' way to generate a linear scale space, based on the essential requirement that new structures must not be created when going from a fine scale to any coarser scale.
Conditions, referred to as ''scale-space axioms
In image processing and computer vision, a scale space framework can be used to represent an image as a family of gradually smoothed images. This framework is very general and a variety of scale space representations exist. A typical approach ...
'', that have been used for deriving the uniqueness of the Gaussian kernel include linearity
Linearity is the property of a mathematical relationship (''function'') that can be graphically represented as a straight line. Linearity is closely related to '' proportionality''. Examples in physics include rectilinear motion, the linear r ...
, shift invariance, semi-group
In mathematics, a semigroup is an algebraic structure consisting of a set together with an associative internal binary operation on it.
The binary operation of a semigroup is most often denoted multiplicatively: ''x''·''y'', or simply ''xy'', ...
structure, non-enhancement of local extrema
In mathematical analysis, the maxima and minima (the respective plurals of maximum and minimum) of a function, known collectively as extrema (the plural of extremum), are the largest and smallest value of the function, either within a given ra ...
, scale invariance
In physics, mathematics and statistics, scale invariance is a feature of objects or laws that do not change if scales of length, energy, or other variables, are multiplied by a common factor, and thus represent a universality.
The technical term ...
and rotational invariance In mathematics, a function defined on an inner product space is said to have rotational invariance if its value does not change when arbitrary rotations are applied to its argument.
Mathematics
Functions
For example, the function
:f(x,y) = x^2 ...
.
In the works, the uniqueness claimed in the arguments based on scale invariance has been criticized, and alternative self-similar scale-space kernels have been proposed. The Gaussian kernel is, however, a unique choice according to the scale-space axiomatics based on causality[ or non-enhancement of local extrema.]
Alternative definition
''Equivalently'', the scale-space family can be defined as the solution of the diffusion equation (for example in terms of the heat equation
In mathematics and physics, the heat equation is a certain partial differential equation. Solutions of the heat equation are sometimes known as caloric functions. The theory of the heat equation was first developed by Joseph Fourier in 1822 for t ...
),
:
with initial condition . This formulation of the scale-space representation ''L'' means that it is possible to interpret the intensity values of the image ''f'' as a "temperature distribution" in the image plane and that the process that generates the scale-space representation as a function of ''t'' corresponds to heat diffusion
Diffusion is the net movement of anything (for example, atoms, ions, molecules, energy) generally from a region of higher concentration to a region of lower concentration. Diffusion is driven by a gradient in Gibbs free energy or chemical p ...
in the image plane over time ''t'' (assuming the thermal conductivity of the material equal to the arbitrarily chosen constant ½). Although this connection may appear superficial for a reader not familiar with differential equation
In mathematics, a differential equation is an equation that relates one or more unknown functions and their derivatives. In applications, the functions generally represent physical quantities, the derivatives represent their rates of change, an ...
s, it is indeed the case that the main scale-space formulation in terms of non-enhancement of local extrema is expressed in terms of a sign condition on partial derivative
In mathematics, a partial derivative of a function of several variables is its derivative with respect to one of those variables, with the others held constant (as opposed to the total derivative, in which all variables are allowed to vary). Part ...
s in the 2+1-D volume generated by the scale space, thus within the framework of partial differential equation
In mathematics, a partial differential equation (PDE) is an equation which imposes relations between the various partial derivatives of a Multivariable calculus, multivariable function.
The function is often thought of as an "unknown" to be sol ...
s. Furthermore, a detailed analysis of the discrete case shows that the diffusion equation provides a unifying link between continuous and discrete scale spaces, which also generalizes to nonlinear scale spaces, for example, using anisotropic diffusion
In image processing and computer vision, anisotropic diffusion, also called Perona–Malik diffusion, is a technique aiming at reducing image noise without removing significant parts of the image content, typically edges, lines or other details ...
. Hence, one may say that the primary way to generate a scale space is by the diffusion equation, and that the Gaussian kernel arises as the Green's function
In mathematics, a Green's function is the impulse response of an inhomogeneous linear differential operator defined on a domain with specified initial conditions or boundary conditions.
This means that if \operatorname is the linear differenti ...
of this specific partial differential equation.
Motivations
The motivation for generating a scale-space representation of a given data set originates from the basic observation that real-world objects are composed of different structures at different scales
Scale or scales may refer to:
Mathematics
* Scale (descriptive set theory), an object defined on a set of points
* Scale (ratio), the ratio of a linear dimension of a model to the corresponding dimension of the original
* Scale factor, a number w ...
. This implies that real-world objects, in contrast to idealized mathematical entities such as points or lines, may appear in different ways depending on the scale of observation.
For example, the concept of a "tree" is appropriate at the scale of meters, while concepts such as leaves and molecules are more appropriate at finer scales.
For a computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the hum ...
system analysing an unknown scene, there is no way to know a priori what scales
Scale or scales may refer to:
Mathematics
* Scale (descriptive set theory), an object defined on a set of points
* Scale (ratio), the ratio of a linear dimension of a model to the corresponding dimension of the original
* Scale factor, a number w ...
are appropriate for describing the interesting structures in the image data.
Hence, the only reasonable approach is to consider descriptions at multiple scales in order to be able to capture the unknown scale variations that may occur.
Taken to the limit, a scale-space representation considers representations at all scales.[
Another motivation to the scale-space concept originates from the process of performing a physical measurement on real-world data. In order to extract any information from a measurement process, one has to apply ''operators of non-infinitesimal size'' to the data. In many branches of computer science and applied mathematics, the size of the measurement operator is disregarded in the theoretical modelling of a problem. The scale-space theory on the other hand explicitly incorporates the need for a non-infinitesimal size of the image operators as an integral part of any measurement as well as any other operation that depends on a real-world measurement.][
There is a close link between scale-space theory and biological vision. Many scale-space operations show a high degree of similarity with receptive field profiles recorded from the mammalian retina and the first stages in the visual cortex.
In these respects, the scale-space framework can be seen as a theoretically well-founded paradigm for early vision, which in addition has been thoroughly tested by algorithms and experiments.][
]
Gaussian derivatives
At any scale in scale space, we can apply local derivative operators to the scale-space representation:
:
Due to the commutative property between the derivative operator and the Gaussian smoothing operator, such ''scale-space derivatives'' can equivalently be computed by convolving the original image with Gaussian derivative operators. For this reason they are often also referred to as ''Gaussian derivatives'':
:
The uniqueness of the Gaussian derivative operators as local operations derived from a scale-space representation can be obtained by similar axiomatic derivations as are used for deriving the uniqueness of the Gaussian kernel for scale-space smoothing.
Visual front end
These Gaussian derivative operators can in turn be combined by linear or non-linear operators into a larger variety of different types of feature detectors, which in many cases can be well modelled by differential geometry
Differential geometry is a mathematical discipline that studies the geometry of smooth shapes and smooth spaces, otherwise known as smooth manifolds. It uses the techniques of differential calculus, integral calculus, linear algebra and multili ...
. Specifically, invariance (or more appropriately ''covariance'') to local geometric transformations, such as rotations or local affine transformations, can be obtained by considering differential invariants under the appropriate class of transformations or alternatively by normalizing the Gaussian derivative operators to a locally determined coordinate frame determined from e.g. a preferred orientation in the image domain, or by applying a preferred local affine transformation to a local image patch (see the article on affine shape adaptation
Affine shape adaptation is a methodology for iteratively adapting the shape of the smoothing kernels in an affine group of smoothing kernels to the local image structure in neighbourhood region of a specific image point. Equivalently, affine shap ...
for further details).
When Gaussian derivative operators and differential invariants are used in this way as basic feature detectors at multiple scales, the uncommitted first stages of visual processing are often referred to as a ''visual front-end''. This overall framework has been applied to a large variety of problems in computer vision, including feature detection, feature classification, image segmentation
In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects ( sets of pixels). The goal of segmentation is to simpli ...
, image matching, motion estimation
Motion estimation is the process of determining ''motion vectors'' that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions ...
, computation of shape
A shape or figure is a graphics, graphical representation of an object or its external boundary, outline, or external Surface (mathematics), surface, as opposed to other properties such as color, Surface texture, texture, or material type.
A pl ...
cues and object recognition
Object recognition – technology in the field of computer vision for finding and identifying objects in an image or video sequence. Humans recognize a multitude of objects in images with little effort, despite the fact that the image of the ...
. The set of Gaussian derivative operators up to a certain order is often referred to as the ''N-jet
An ''N''-jet is the set of (partial) derivatives of a function f(x) up to order ''N''.
Specifically, in the area of computer vision, the ''N''-jet is usually computed from a scale space representation L of the input image f(x, y), and the p ...
'' and constitutes a basic type of feature within the scale-space framework.
Detector examples
Following the idea of expressing visual operations in terms of differential invariants computed at multiple scales using Gaussian derivative operators, we can express an edge detector from the set of points that satisfy the requirement that the gradient magnitude
:
should assume a local maximum in the gradient direction
:
By working out the differential geometry, it can be shown [ that this differential edge detector can equivalently be expressed from the zero-crossings of the second-order differential invariant
:
that satisfy the following sign condition on a third-order differential invariant:
:
Similarly, multi-scale blob detectors at any given fixed scale][ can be obtained from local maxima and local minima of either the ]Laplacian
In mathematics, the Laplace operator or Laplacian is a differential operator given by the divergence of the gradient of a scalar function on Euclidean space. It is usually denoted by the symbols \nabla\cdot\nabla, \nabla^2 (where \nabla is the ...
operator (also referred to as the Laplacian of Gaussian)
:
or the determinant of the Hessian matrix
:
In an analogous fashion, corner detectors and ridge and valley detectors can be expressed as local maxima, minima or zero-crossings of multi-scale differential invariants defined from Gaussian derivatives. The algebraic expressions for the corner and ridge detection operators are, however, somewhat more complex and the reader is referred to the articles on corner detection
Corner detection is an approach used within computer vision systems to extract certain kinds of features and infer the contents of an image. Corner detection is frequently used in motion detection, image registration, video tracking, image mo ...
and ridge detection
In image processing, ridge detection is the attempt, via software, to locate ridges in an image, defined as curves whose points are local maxima of the function, akin to geographical ridges.
For a function of ''N'' variables, its ridges are ...
for further details.
Scale-space operations have also been frequently used for expressing coarse-to-fine methods, in particular for tasks such as image matching and for multi-scale image segmentation.
Scale selection
The theory presented so far describes a well-founded framework for ''representing'' image structures at multiple scales. In many cases it is, however, also necessary to select locally appropriate scales for further analysis. This need for ''scale selection'' originates from two major reasons; (i) real-world objects may have different size, and this size may be unknown to the vision system, and (ii) the distance between the object and the camera can vary, and this distance information may also be unknown ''a priori''.
A highly useful property of scale-space representation is that image representations can be made invariant to scales, by performing automatic local scale selection[T. Lindeberg "Spatio-temporal scale selection in video data", Journal of Mathematical Imaging and Vision, 60(4): 525–562, 2018.]
/ref>[T. Lindeberg "Dense scale selection over space, time and space-time", SIAM Journal on Imaging Sciences, 11(1): 407–441, 2018.]
/ref> based on local maxima (or minima) over scales of scale-normalized derivative
In mathematics, the derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value). Derivatives are a fundamental tool of calculus. F ...
s
:
where