|
Pinsker's Inequality
In information theory, Pinsker's inequality, named after its inventor Mark Semenovich Pinsker, is an inequality that bounds the total variation distance (or statistical distance) in terms of the Kullback–Leibler divergence. The inequality is tight up to constant factors. Formal statement Pinsker's inequality states that, if P and Q are two probability distributions on a measurable space (X, \Sigma), then :\delta(P,Q) \le \sqrt, where :\delta(P,Q)=\sup \bigl\ is the total variation distance (or statistical distance) between P and Q and :D_(P\parallel Q) = \operatorname_P \left( \log \frac \right) = \int_X \left( \log \frac \right) \, \mathrm P is the Kullback–Leibler divergence in nats. When the sample space X is a finite set, the Kullback–Leibler divergence is given by : D_(P\parallel Q) = \sum_ \left( \log \frac\right) P(i)\! Note that in terms of the total variation norm \, P - Q \, of the signed measure P - Q, Pinsker's inequality differs from the one given a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Information Theory
Information theory is the scientific study of the quantification (science), quantification, computer data storage, storage, and telecommunication, communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. The field is at the intersection of probability theory, statistics, computer science, statistical mechanics, information engineering (field), information engineering, and electrical engineering. A key measure in information theory is information entropy, entropy. Entropy quantifies the amount of uncertainty involved in the value of a random variable or the outcome of a random process. For example, identifying the outcome of a fair coin flip (with two equally likely outcomes) provides less information (lower entropy) than specifying the outcome from a roll of a dice, die (with six equally likely outcomes). Some other important measures in information theory are mutual informat ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
F-divergence
In probability theory, an f-divergence is a function D_f(P\, Q) that measures the difference between two probability distributions P and Q. Many common divergences, such as KL-divergence, Hellinger distance, and total variation distance, are special cases of f-divergence. History These divergences were introduced by Alfréd Rényi in the same paper where he introduced the well-known Rényi entropy. He proved that these divergences decrease in Markov processes. ''f''-divergences were studied further independently by , and and are sometimes known as Csiszár f-divergences, Csiszár-Morimoto divergences, or Ali-Silvey distances. Definition Non-singular case Let P and Q be two probability distributions over a space \Omega, such that P\ll Q, that is, P is absolutely continuous with respect to Q. Then, for a convex function f: , \infty)\to(-\infty, \infty/math> such that f(x) is finite for all x > 0, f(1)=0, and f(0)=\lim_ f(t) (which could be infinite), the f-divergence ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Support (measure Theory)
In mathematics, the support (sometimes topological support or spectrum) of a measure ''μ'' on a measurable topological space (''X'', Borel(''X'')) is a precise notion of where in the space ''X'' the measure "lives". It is defined to be the largest ( closed) subset of ''X'' for which every open neighbourhood of every point of the set has positive measure. Motivation A (non-negative) measure \mu on a measurable space (X, \Sigma) is really a function \mu : \Sigma \to , +\infty. Therefore, in terms of the usual definition of support, the support of \mu is a subset of the σ-algebra \Sigma : :\operatorname (\mu) := \overline, where the overbar denotes set closure. However, this definition is somewhat unsatisfactory: we use the notion of closure, but we do not even have a topology on \Sigma . What we really want to know is where in the space X the measure \mu is non-zero. Consider two examples: # Lebesgue measure \lambda on the real line \mathbb . It seems ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Johannes Kemperman
Johannes Henricus Bernardus Kemperman (July 16, 1924 – June 13, 2011) was a Dutch mathematician. He taught at the University of Rochester for 25 years, and also worked at Purdue University and Rutgers University for ten years, each. Born in Amsterdam, he received his education from the University of Amsterdam The University of Amsterdam (abbreviated as UvA, nl, Universiteit van Amsterdam) is a public research university located in Amsterdam, Netherlands. The UvA is one of two large, publicly funded research universities in the city, the other being .... Selected publications * * * References External links * {{DEFAULTSORT:Kemperman, Johannes 1924 births 2011 deaths Dutch mathematicians University of Amsterdam alumni Purdue University faculty University of Rochester faculty Rutgers University faculty Dutch expatriates in the United States Scientists from Amsterdam ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Imre Csiszár
Imre Csiszár () is a Hungarian mathematician with contributions to information theory and probability theory. In 1996 he won the Claude E. Shannon Award, the highest annual award given in the field of information theory. He was born on February 7, 1938, in Miskolc, Hungary. He became interested in mathematics in middle school. He was inspired by his father who was a forest engineer and was among the first to use mathematical techniques in his area. He studied mathematics at the Eötvös Loránd University, Budapest, and received his Diploma in 1961. He got his PhD in 1967 and the scientific degree Doctor of Mathematical Science in 1977. Later, he was influenced by Alfréd Rényi, who was very active in the area of probability theory. In 1990 he was elected Corresponding Member of the Hungarian Academy of Sciences, and in 1995 he became Full Member. Professor Csiszar has been with the Mathematical Institute of the Hungarian Academy of Sciences since 1961. He has been Head of ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Solomon Kullback
Solomon Kullback (April 3, 1907August 5, 1994) was an American cryptanalyst and mathematician, who was one of the first three employees hired by William F. Friedman at the US Army's Signal Intelligence Service (SIS) in the 1930s, along with Frank Rowlett and Abraham Sinkov. He went on to a long and distinguished career at SIS and its eventual successor, the National Security Agency (NSA). Kullback was the Chief Scientist at the NSA until his retirement in 1962, whereupon he took a position at the George Washington University. The Kullback–Leibler divergence is named after Kullback and Richard Leibler. Life and career Kullback was born to Jewish parents in Brooklyn, New York. His father Nathan had been born in Vilna, Russian Empire, (now Vilnius, Lithuania) and had immigrated to the US as a young man circa 1905, and became a naturalized American in 1911. Kullback attended Boys High School in Brooklyn. He then went to City College of New York, graduating with a BA in 1927 and an ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Bretagnolle–Huber Inequality
In information theory, the Bretagnolle–Huber inequality bounds the total variation distance between two probability distributions P and Q by a concave and bounded function of the Kullback–Leibler divergence D_\mathrm(P \parallel Q). The bound can be viewed as an alternative to the well-known Pinsker's inequality: when D_\mathrm(P \parallel Q) is large (larger than 2 for instance.), Pinsker's inequality is vacuous, while Bretagnolle–Huber remains bounded and hence non-vacuous. It is used in statistics and machine learning to prove information-theoretic lower bounds relying on hypothesis testing Formal statement Preliminary definitions Let P and Q be two probability distributions on a measurable space (\mathcal, \mathcal). Recall that the total variation between P and Q is defined by : d_\mathrm(P,Q) = \sup_ \. The Kullback-Leibler divergence is defined as follows: :D_\mathrm(P \parallel Q) = \begin \int_ \log\bigl(\frac\bigr)\, dP & \text P \ll Q, \\ mm+\infty & \tex ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Sedrakyan's Inequality
The following inequality is known as Sedrakyan's inequality, Bergström's inequality, Engel's form or Titu's lemma, respectively, referring to the article ''About the applications of one useful inequality'' of Nairi Sedrakyan published in 1997, to the book ''Problem-solving strategies'' of Arthur Engel published in 1998 and to the book ''Mathematical Olympiad Treasures'' of Titu Andreescu published in 2003. It is a direct consequence of Cauchy–Bunyakovsky–Schwarz inequality. Nevertheless, in his article (1997) Sedrakyan has noticed that written in this form this inequality can be used as a mathematical proof technique and it has very useful new applications. In the book ''Algebraic Inequalities'' (Sedrakyan) are provided several generalizations of this inequality. Statement of the inequality For any reals a_1, a_2, a_3, \ldots, a_n and positive reals b_1, b_2, b_3,\ldots, b_n, we have \frac + \frac + \cdots + \frac \geq \frac. (Nairi Sedrakyan (1997), Arthur Engel (199 ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
John Pollard (mathematician)
John M. Pollard (born 1941) is a British mathematician who has invented algorithms for the factorization of large numbers and for the calculation of discrete logarithms. His factorization algorithms include the rho, ''p'' − 1, and the first version of the special number field sieve, which has since been improved by others. His discrete logarithm algorithms include the rho algorithm for logarithms and the kangaroo algorithm. He received the RSA Award for Excellence in Mathematics RSA may refer to: Organizations Academia and education * Rabbinical Seminary of America, a yeshiva in New York City *Regional Science Association International (formerly the Regional Science Association), a US-based learned society *Renaissance S .... External links John Pollard's web site Living people 20th-century British mathematicians 21st-century British mathematicians Number theorists Place of birth missing (living people) 1941 births {{UK-mathematician-stub ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Probability Density Function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can be interpreted as providing a ''relative likelihood'' that the value of the random variable would be close to that sample. Probability density is the probability per unit length, in other words, while the ''absolute likelihood'' for a continuous random variable to take on any particular value is 0 (since there is an infinite set of possible values to begin with), the value of the PDF at two different samples can be used to infer, in any particular draw of the random variable, how much more likely it is that the random variable would be close to one sample compared to the other sample. In a more precise sense, the PDF is used to specify the probability of the random variable falling ''within a particular range of values'', as opposed to ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Total Variation
In mathematics, the total variation identifies several slightly different concepts, related to the (local or global) structure of the codomain of a function or a measure. For a real-valued continuous function ''f'', defined on an interval 'a'', ''b''⊂ R, its total variation on the interval of definition is a measure of the one-dimensional arclength of the curve with parametric equation ''x'' ↦ ''f''(''x''), for ''x'' ∈ 'a'', ''b'' Functions whose total variation is finite are called functions of bounded variation. Historical note The concept of total variation for functions of one real variable was first introduced by Camille Jordan in the paper . He used the new concept in order to prove a convergence theorem for Fourier series of discontinuous periodic functions whose variation is bounded. The extension of the concept to functions of more than one variable however is not simple for various reasons. Definitions Total variation for functions of one real variable Th ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Partition Inequality
Partition may refer to: Computing Hardware * Disk partitioning, the division of a hard disk drive * Memory partition, a subdivision of a computer's memory, usually for use by a single job Software * Partition (database), the division of a database * Logical partition (LPAR), a subset of a computer's resources, virtualized as a separate computer Problems * Binary space partitioning * Partition problem, an NP-complete problem in computer science Mathematics * Partition (number theory), a way to write a number as a sum of other numbers * Multiplicative partition, a way to write a number as a product of other numbers * Partition of an interval * Partition of a set * Partition of unity, a certain kind of set of functions on a topological space * Plane partition * Graph partition Natural science * Partition function (quantum field theory) * Partition function (statistical mechanics) * Partition coefficient, a concept in organic chemistry Law and politics * Partition (law), ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
