In
probability theory
Probability theory is the branch of mathematics concerned with probability. Although there are several different probability interpretations, probability theory treats the concept in a rigorous mathematical manner by expressing it through a set o ...
, a Markov reward model or Markov reward process is a stochastic process which extends either a
Markov chain
A Markov chain or Markov process is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Informally, this may be thought of as, "What happe ...
or
continuous-time Markov chain
A continuous-time Markov chain (CTMC) is a continuous stochastic process in which, for each state, the process will change state according to an exponential random variable and then move to a different state as specified by the probabilities of ...
by adding a reward rate to each state. An additional variable records the reward accumulated up to the current time. Features of interest in the model include expected reward at a given time and expected time to accumulate a given reward. The model appears in
Ronald A. Howard Ronald Arthur Howard (born August 27, 1934) is an emeritus professor in the Department of Engineering-Economic Systems (now the Department of Management Science and Engineering) in the School of Engineering at Stanford University.
Howard directs t ...
's book. The models are often studied in the context of
Markov decision processes where a decision strategy can impact the rewards received.
The Markov Reward Model Checker tool can be used to numerically compute transient and stationary properties of Markov reward models.
Continuous-time Markov chain
The accumulated reward at a time ''t'' can be computed numerically over the time domain or by evaluating the linear hyperbolic system of equations which describe the accumulated reward using transform methods or finite difference methods.
See also
*
Markov chain Monte Carlo
In statistics, Markov chain Monte Carlo (MCMC) methods comprise a class of algorithms for sampling from a probability distribution. By constructing a Markov chain that has the desired distribution as its equilibrium distribution, one can obtain ...
References
Markov processes
{{probability-stub