Probability theory is the branch of
mathematics concerned with
probability. Although there are several different
probability interpretations
The word probability has been used in a variety of ways since it was first applied to the mathematical study of games of chance. Does probability measure the real, physical, tendency of something to occur, or is it a measure of how strongly one b ...
, probability theory treats the concept in a rigorous mathematical manner by expressing it through a set of
axioms
An axiom, postulate, or assumption is a statement that is taken to be true, to serve as a premise or starting point for further reasoning and arguments. The word comes from the Ancient Greek word (), meaning 'that which is thought worthy or f ...
. Typically these axioms formalise probability in terms of a
probability space
In probability theory, a probability space or a probability triple (\Omega, \mathcal, P) is a mathematical construct that provides a formal model of a random process or "experiment". For example, one can define a probability space which models t ...
, which assigns a
measure taking values between 0 and 1, termed the
probability measure, to a set of outcomes called the
sample space
In probability theory, the sample space (also called sample description space, possibility space, or outcome space) of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually den ...
. Any specified subset of the sample space is called an
event.
Central subjects in probability theory include discrete and continuous
random variables,
probability distributions
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
, and
stochastic processes (which provide mathematical abstractions of
non-deterministic or uncertain processes or measured
quantities that may either be single occurrences or evolve over time in a random fashion).
Although it is not possible to perfectly predict random events, much can be said about their behavior. Two major results in probability theory describing such behaviour are the
law of large numbers and the
central limit theorem
In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themsel ...
.
As a mathematical foundation for
statistics, probability theory is essential to many human activities that involve quantitative analysis of data. Methods of probability theory also apply to descriptions of complex systems given only partial knowledge of their state, as in
statistical mechanics or
sequential estimation. A great discovery of twentieth-century
physics
Physics is the natural science that studies matter, its fundamental constituents, its motion and behavior through space and time, and the related entities of energy and force. "Physical science is that department of knowledge which r ...
was the probabilistic nature of physical phenomena at atomic scales, described in
quantum mechanics
Quantum mechanics is a fundamental theory in physics that provides a description of the physical properties of nature at the scale of atoms and subatomic particles. It is the foundation of all quantum physics including quantum chemistr ...
.
History of probability
The modern mathematical theory of
probability has its roots in attempts to analyze
games of chance by
Gerolamo Cardano in the sixteenth century, and by
Pierre de Fermat and
Blaise Pascal in the seventeenth century (for example the "
problem of points
The problem of points, also called the problem of division of the stakes, is a classical problem in probability theory. One of the famous problems that motivated the beginnings of modern probability theory in the 17th century, it led Blaise Pascal ...
").
Christiaan Huygens published a book on the subject in 1657. In the 19th century, what is considered the
classical definition of probability The classical definition or interpretation of probability is identified with the works of Jacob Bernoulli and Pierre-Simon Laplace. As stated in Laplace's ''Théorie analytique des probabilités'',
:The probability of an event is the ratio of the n ...
was completed by
Pierre Laplace
Pierre-Simon, marquis de Laplace (; ; 23 March 1749 – 5 March 1827) was a French scholar and polymath whose work was important to the development of engineering, mathematics, statistics, physics, astronomy, and philosophy. He summarized ...
.
Initially, probability theory mainly considered events, and its methods were mainly
combinatorial
Combinatorics is an area of mathematics primarily concerned with counting, both as a means and an end in obtaining results, and certain properties of finite structures. It is closely related to many other areas of mathematics and has many ap ...
. Eventually,
analytical considerations compelled the incorporation of variables into the theory.
This culminated in modern probability theory, on foundations laid by
Andrey Nikolaevich Kolmogorov. Kolmogorov combined the notion of
sample space
In probability theory, the sample space (also called sample description space, possibility space, or outcome space) of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually den ...
, introduced by
Richard von Mises, and
measure theory and presented his
axiom system for probability theory in 1933. This became the mostly undisputed
axiomatic basis for modern probability theory; but, alternatives exist, such as the adoption of finite rather than countable additivity by
Bruno de Finetti
Bruno de Finetti (13 June 1906 – 20 July 1985) was an Italian probabilist statistician and actuary, noted for the "operational subjective" conception of probability. The classic exposition of his distinctive theory is the 1937 "La prévision: ...
.
Treatment
Most introductions to probability theory treat discrete probability distributions and continuous probability distributions separately. The measure theory-based treatment of probability covers the discrete, continuous, a mix of the two, and more.
Motivation
Consider an
experiment that can produce a number of outcomes. The set of all outcomes is called the ''
sample space
In probability theory, the sample space (also called sample description space, possibility space, or outcome space) of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually den ...
'' of the experiment. The ''
power set
In mathematics, the power set (or powerset) of a set is the set of all subsets of , including the empty set and itself. In axiomatic set theory (as developed, for example, in the ZFC axioms), the existence of the power set of any set is post ...
'' of the sample space (or equivalently, the event space) is formed by considering all different collections of possible results. For example, rolling an honest die produces one of six possible results. One collection of possible results corresponds to getting an odd number. Thus, the subset is an element of the power set of the sample space of die rolls. These collections are called ''events''. In this case, is the event that the die falls on some odd number. If the results that actually occur fall in a given event, that event is said to have occurred.
Probability is a
way of assigning every "event" a value between zero and one, with the requirement that the event made up of all possible results (in our example, the event ) be assigned a value of one. To qualify as a
probability distribution, the assignment of values must satisfy the requirement that if you look at a collection of mutually exclusive events (events that contain no common results, e.g., the events , , and are all mutually exclusive), the probability that any of these events occurs is given by the sum of the probabilities of the events.
The probability that any one of the events , , or will occur is 5/6. This is the same as saying that the probability of event is 5/6. This event encompasses the possibility of any number except five being rolled. The mutually exclusive event has a probability of 1/6, and the event has a probability of 1, that is, absolute certainty.
When doing calculations using the outcomes of an experiment, it is necessary that all those
elementary events have a number assigned to them. This is done using a
random variable. A random variable is a function that assigns to each elementary event in the sample space a
real number. This function is usually denoted by a capital letter. In the case of a die, the assignment of a number to a certain elementary events can be done using the
identity function. This does not always work. For example, when
flipping a coin the two possible outcomes are "heads" and "tails". In this example, the random variable ''X'' could assign to the outcome "heads" the number "0" (
) and to the outcome "tails" the number "1" (
).
Discrete probability distributions
deals with events that occur in
countable
In mathematics, a set is countable if either it is finite or it can be made in one to one correspondence with the set of natural numbers. Equivalently, a set is ''countable'' if there exists an injective function from it into the natural numbers ...
sample spaces.
Examples: Throwing
dice, experiments with
decks of cards,
random walk, and tossing
coins
:
Initially the probability of an event to occur was defined as the number of cases favorable for the event, over the number of total outcomes possible in an equiprobable sample space: see
Classical definition of probability The classical definition or interpretation of probability is identified with the works of Jacob Bernoulli and Pierre-Simon Laplace. As stated in Laplace's ''Théorie analytique des probabilités'',
:The probability of an event is the ratio of the n ...
.
For example, if the event is "occurrence of an even number when a die is rolled", the probability is given by
, since 3 faces out of the 6 have even numbers and each face has the same probability of appearing.
:
The modern definition starts with a
finite or countable set called the
sample space
In probability theory, the sample space (also called sample description space, possibility space, or outcome space) of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually den ...
, which relates to the set of all ''possible outcomes'' in classical sense, denoted by
. It is then assumed that for each element
, an intrinsic "probability" value
is attached, which satisfies the following properties:
#
#
That is, the probability function ''f''(''x'') lies between zero and one for every value of ''x'' in the sample space ''Ω'', and the sum of ''f''(''x'') over all values ''x'' in the sample space ''Ω'' is equal to 1. An is defined as any
subset of the sample space
. The of the event
is defined as
:
So, the probability of the entire sample space is 1, and the probability of the null event is 0.
The function
mapping a point in the sample space to the "probability" value is called a abbreviated as . The modern definition does not try to answer how probability mass functions are obtained; instead, it builds a theory that assumes their existence.
Continuous probability distributions
deals with events that occur in a continuous sample space.
:
The classical definition breaks down when confronted with the continuous case. See
Bertrand's paradox.
:
If the sample space of a random variable ''X'' is the set of
real numbers (
) or a subset thereof, then a function called the (or )
exists, defined by
. That is, ''F''(''x'') returns the probability that ''X'' will be less than or equal to ''x''.
The cdf necessarily satisfies the following properties.
#
is a
monotonically non-decreasing,
right-continuous
In mathematics, a continuous function is a function such that a continuous variation (that is a change without jump) of the argument induces a continuous variation of the value of the function. This means that there are no abrupt changes in valu ...
function;
#
#
If
is
absolutely continuous
In calculus, absolute continuity is a smoothness property of functions that is stronger than continuity and uniform continuity. The notion of absolute continuity allows one to obtain generalizations of the relationship between the two central ope ...
, i.e., its derivative exists and integrating the derivative gives us the cdf back again, then the random variable ''X'' is said to have a or or simply
For a set
, the probability of the random variable ''X'' being in
is
:
In case the probability density function exists, this can be written as
:
Whereas the ''pdf'' exists only for continuous random variables, the ''cdf'' exists for all random variables (including discrete random variables) that take values in
These concepts can be generalized for
multidimensional cases on
and other continuous sample spaces.
Measure-theoretic probability theory
The ''
raison d'être
Raison d'être is a French expression commonly used in English, meaning "reason for being" or "reason to be".
Raison d'être may refer to:
Music
* Raison d'être (band), a Swedish dark-ambient-industrial-drone music project
* ''Raison D'être' ...
'' of the measure-theoretic treatment of probability is that it unifies the discrete and the continuous cases, and makes the difference a question of which measure is used. Furthermore, it covers distributions that are neither discrete nor continuous nor mixtures of the two.
An example of such distributions could be a mix of discrete and continuous distributions—for example, a random variable that is 0 with probability 1/2, and takes a random value from a normal distribution with probability 1/2. It can still be studied to some extent by considering it to have a pdf of
, where