HOME

TheInfoList



OR:

In
numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of ...
, the order of convergence and the rate of convergence of a
convergent sequence As the positive integer n becomes larger and larger, the value n\cdot \sin\left(\tfrac1\right) becomes arbitrarily close to 1. We say that "the limit of the sequence n\cdot \sin\left(\tfrac1\right) equals 1." In mathematics, the limit ...
are quantities that represent how quickly the sequence approaches its limit. A sequence (x_n) that converges to x^* is said to have ''order of convergence'' q \geq 1 and ''rate of convergence'' \mu if : \lim _ \frac=\mu. The rate of convergence \mu is also called the ''asymptotic error constant''. Note that this terminology is not standardized and some authors will use ''rate'' where this article uses ''order'' (e.g., ). In practice, the rate and order of convergence provide useful insights when using
iterative methods In computational mathematics, an iterative method is a mathematical procedure that uses an initial value to generate a sequence of improving approximate solutions for a class of problems, in which the ''n''-th approximation is derived from the pre ...
for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the
asymptotic behavior In mathematical analysis, asymptotic analysis, also known as asymptotics, is a method of describing limiting behavior. As an illustration, suppose that we are interested in the properties of a function as becomes very large. If , then as beco ...
of a sequence does not give conclusive information about any finite part of the sequence. Similar concepts are used for
discretization In applied mathematics, discretization is the process of transferring continuous functions, models, variables, and equations into discrete counterparts. This process is usually carried out as a first step toward making them suitable for numerical ...
methods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology, in this case, is different from the terminology for iterative methods.
Series acceleration In mathematics, series acceleration is one of a collection of sequence transformations for improving the rate of convergence of a series. Techniques for series acceleration are often applied in numerical analysis, where they are used to improve the ...
is a collection of techniques for improving the rate of convergence of a series discretization. Such acceleration is commonly accomplished with
sequence transformation In mathematics, a sequence transformation is an operator acting on a given space of sequences (a sequence space). Sequence transformations include linear mappings such as convolution with another sequence, and resummation of a sequence and, more ...
s.


Convergence speed for iterative methods


Q-convergence definitions

Suppose that the
sequence In mathematics, a sequence is an enumerated collection of objects in which repetitions are allowed and order matters. Like a set, it contains members (also called ''elements'', or ''terms''). The number of elements (possibly infinite) is calle ...
(x_k) converges to the number L. The sequence is said to ''converge Q-linearly to L'' if there exists a number \mu \in (0, 1) such that : \lim_ \frac = \mu. The number \mu is called the ''rate of convergence.'' The sequence is said to ''converge Q-superlinearly to L'' (i.e. faster than linearly) in all the cases where q > 1 and also the case q = 1, \mu = 0 if : \lim_ \frac = \mu. The sequence is said to ''converge Q-sublinearly to L'' (i.e. slower than linearly) if : \lim_ \frac = 1. The sequence (x_k) ''converges logarithmically to L'' if the sequence converges sublinearly and additionally if :\lim_ \frac = 1. Note that unlike previous definitions, logarithmic convergence is not called "Q-logarithmic." In order to further classify convergence, the ''order of convergence'' is defined as follows. The sequence is said to ''converge with order q to L'' for q \geq 1 if :\lim_ \frac = M for some positive constant 0 < M < \infty if q > 1, and 0 < M < 1 if q = 1. In particular, convergence with order * q = 1 is called ''linear convergence'' (if M<1), * q = 2 is called ''quadratic convergence'', * q = 3 is called ''cubic convergence'', * etc. Some sources require that q is strictly greater than 1 since the q=1 case requires M<1 so is best treated separately. It is not necessary, however, that q be an integer. For example, the
secant method In numerical analysis, the secant method is a root-finding algorithm that uses a succession of roots of secant lines to better approximate a root of a function ''f''. The secant method can be thought of as a finite-difference approximation of ...
, when converging to a regular,
simple root Simple or SIMPLE may refer to: *Simplicity, the state or quality of being simple Arts and entertainment * ''Simple'' (album), by Andy Yorke, 2008, and its title track * "Simple" (Florida Georgia Line song), 2018 * "Simple", a song by Johnn ...
, has an order of φ ≈ 1.618. In the definitions above, the "Q-" stands for "quotient" because the terms are defined using the quotient between two successive terms. Often, however, the "Q-" is dropped and a sequence is simply said to have ''linear convergence'', ''quadratic convergence'', etc.


Order estimation

A practical method to calculate the order of convergence for a sequence is to calculate the following sequence, which converges to q :q \approx \frac.


R-convergence definition

The Q-convergence definitions have a shortcoming in that they do not include some sequences, such as the sequence (b_k) below, which converge reasonably fast, but whose rate is variable. Therefore, the definition of rate of convergence is extended as follows. Suppose that (x_k) converges to L. The sequence is said to ''converge R-linearly to L'' if there exists a sequence (\varepsilon_k) such that : , x_k - L, \le\varepsilon_k\quad\textk \,, and (\varepsilon_k) converges Q-linearly to zero. The "R-" prefix stands for "root".


Examples

Consider the sequence :(a_k) = \left\. It can be shown that this sequence converges to L = 0. To determine the type of convergence, we plug the sequence into the definition of Q-linear convergence, :\lim_ \frac = \lim_ \frac = \frac. Thus, we find that (a_k) converges Q-linearly and has a convergence rate of \mu = 1/2. More generally, for any c \in \mathbb, \mu \in (-1, 1), the sequence (c\mu^k) converges linearly with rate , \mu, . The sequence :(b_k) = \left\ also converges linearly to 0 with rate 1/2 under the R-convergence definition, but not under the Q-convergence definition. (Note that \lfloor x \rfloor is the
floor function In mathematics and computer science, the floor function is the function that takes as input a real number , and gives as output the greatest integer less than or equal to , denoted or . Similarly, the ceiling function maps to the least int ...
, which gives the largest integer that is less than or equal to x.) The sequence :(c_k) = \left\ converges superlinearly. In fact, it is quadratically convergent. Finally, the sequence :(d_k) = \left\ converges sublinearly and logarithmically.


Convergence speed for discretization methods

A similar situation exists for discretization methods designed to approximate a function y = f(x), which might be an integral being approximated by
numerical quadrature In analysis, numerical integration comprises a broad family of algorithms for calculating the numerical value of a definite integral, and by extension, the term is also sometimes used to describe the numerical solution of differential equations ...
, or the solution of an ordinary differential equation (see example below). The discretization method generates a sequence , where each successive y_j is a function of y_,y_,... along with the grid spacing h between successive values of the independent variable x. The important parameter here for the convergence speed to y = f(x) is the grid spacing h, inversely proportional to the number of grid points, i.e. the number of points in the sequence required to reach a given value of x. In this case, the sequence (y_n) is said to converge to the sequence f(x_n) with order ''q'' if there exists a constant ''C'' such that : , y_n - f(x_n), < C h^ \text n. This is written as , y_n - f(x_n), = \mathcal(h^) using
big O notation Big ''O'' notation is a mathematical notation that describes the limiting behavior of a function when the argument tends towards a particular value or infinity. Big O is a member of a family of notations invented by Paul Bachmann, Edmund Lan ...
. This is the relevant definition when discussing methods for
numerical quadrature In analysis, numerical integration comprises a broad family of algorithms for calculating the numerical value of a definite integral, and by extension, the term is also sometimes used to describe the numerical solution of differential equations ...
or the solution of ordinary differential equations. A practical method to estimate the order of convergence for a discretization method is pick step sizes h_\text and h_\text and calculate the resulting errors e_\text and e_\text. The order of convergence is then approximated by the following formula: :q \approx \frac, which comes from writing the truncation error, at the old and new grid spacings, as : e = , y_n - f(x_n), = \mathcal(h^). The error e is, more specifically, a global truncation error (GTE), in that it represents a sum of errors accumulated over all n iterations, as opposed to a local truncation error (LTE) over just one iteration.


Example of discretization methods

Consider the ordinary differential equation : \frac = -\kappa y with initial condition y(0) = y_0. We can solve this equation using the Forward Euler scheme for numerical discretization: : \frac = -\kappa y_, which generates the sequence : y_ = y_n(1 - h\kappa). In terms of y(0) = y_0, this sequence is as follows, from the
Binomial theorem In elementary algebra, the binomial theorem (or binomial expansion) describes the algebraic expansion of powers of a binomial. According to the theorem, it is possible to expand the polynomial into a sum involving terms of the form , where the ...
: : y_ = y_0(1 - h\kappa)^n = y_0\left(1 - nh\kappa + n(n-1)\frac + ....\right). The exact solution to this ODE is y = f(x) = y_0\exp(-\kappa x), corresponding to the following
Taylor expansion In mathematics, the Taylor series or Taylor expansion of a function is an infinite sum of terms that are expressed in terms of the function's derivatives at a single point. For most common functions, the function and the sum of its Taylor serie ...
in h\kappa for h\kappa \ll 1: : f(x_n) = f(nh) = y_0\exp(-\kappa nh) = y_0\left exp(-\kappa h)\rightn = y_0\left(1 - h\kappa + \frac + ....\right)^n = y_0\left(1 - nh\kappa + \frac + ...\right). In this case, the truncation error is : e = , y_n - f(x_n), = \frac = \mathcal(h^), so (y_n) converges to f(x_n) with a convergence rate q = 2.


Examples (continued)

The sequence (d_k) with d_k = 1/(k+1) was introduced above. This sequence converges with order 1 according to the convention for discretization methods. The sequence (a_k) with a_k = 2^, which was also introduced above, converges with order ''q'' for every number ''q''. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.


Recurrent sequences and fixed points

The case of recurrent sequences x_:=f(x_n) which occurs in
dynamical systems In mathematics, a dynamical system is a system in which a function describes the time dependence of a point in an ambient space. Examples include the mathematical models that describe the swinging of a clock pendulum, the flow of water in a p ...
and in the context of various
fixed point theorems Fixed may refer to: * ''Fixed'' (EP), EP by Nine Inch Nails * ''Fixed'', an upcoming 2D adult animated film directed by Genndy Tartakovsky * Fixed (typeface), a collection of monospace bitmap fonts that is distributed with the X Window System * F ...
is of particular interest. Assuming that the relevant derivatives of ''f'' are continuous, one can (easily) show that for a fixed point f(p)=p such that , f'(p), < 1, one has at linear convergence for any starting value x_0 sufficiently close to ''p''. If , f'(p), = 0 and , f''(p), < 1, then one has at least quadratic convergence, and so on. If , f'(p), > 1, then one has a
repulsive fixed point A fixed point (sometimes shortened to fixpoint, also known as an invariant point) is a value that does not change under a given transformation. Specifically, in mathematics, a fixed point of a function is an element that is mapped to itself by the ...
and no starting value will produce a sequence converging to ''p'' (unless one directly jumps to the point ''p'' itself).


Acceleration of convergence

Many methods exist to increase the rate of convergence of a given sequence, i.e. to transform a given sequence into one converging faster to the same limit. Such techniques are in general known as "
series acceleration In mathematics, series acceleration is one of a collection of sequence transformations for improving the rate of convergence of a series. Techniques for series acceleration are often applied in numerical analysis, where they are used to improve the ...
". The goal of the transformed sequence is to reduce the
computational cost In computational complexity theory, a computational resource is a resource used by some computational models in the solution of computational problems. The simplest computational resources are computation time, the number of steps necessary to s ...
of the calculation. One example of series acceleration is
Aitken's delta-squared process In numerical analysis, Aitken's delta-squared process or Aitken extrapolation is a series acceleration method, used for accelerating the rate of convergence of a sequence. It is named after Alexander Aitken, who introduced this method in 1926.Alexa ...
. (It should be noted, though, that these methods in general (and in particular Aitken's method) do not increase the order of convergence, and are useful only if initially the convergence is not faster than linear: If (x_n) convergences linearly, one gets a sequence (a_n) that still converges linearly (except for pathologically designed special cases), but faster in the sense that \lim (a_n-L)/(x_n-L)= 0. On the other hand, if the convergence is already of order ≥ 2, Aitken's method will bring no improvement.)


References


Literature

The simple definition is used in *
Michelle Schatzman Michelle Schatzman (1949–2010) was a French mathematician, specializing in applied mathematics, who combined research as a CNRS research director and teaching as a professor at the Claude Bernard University Lyon 1. Biography Michelle Véra Scha ...
(2002), ''Numerical analysis: a mathematical introduction'', Clarendon Press, Oxford. . The extended definition is used in * Walter Gautschi (1997), ''Numerical analysis: an introduction,'' Birkhäuser, Boston. . *
Endre Süli Endre Süli (also, Endre Suli or Endre Šili) is a mathematician. He is Professor of Numerical Analysis in the Mathematical Institute, University of Oxford, Fellow and Tutor in Mathematics at Worcester College, Oxford and Adjunct Fellow of Li ...
and David Mayers (2003), ''An introduction to numerical analysis,'' Cambridge University Press. . The Big O definition is used in *Richard L. Burden and J. Douglas Faires (2001), ''Numerical Analysis'' (7th ed.), Brooks/Cole. The terms ''Q-linear'' and ''R-linear'' are used in; The Big O definition when using Taylor series is used in * . {{Differential equations topics Numerical analysis
Convergence Convergence may refer to: Arts and media Literature *''Convergence'' (book series), edited by Ruth Nanda Anshen * "Convergence" (comics), two separate story lines published by DC Comics: **A four-part crossover storyline that united the four Wei ...