Transition probabilities

The probabilities of transition of a Markov chain $\xi ( t)$ from a state $i$ into a state $j$ in a time interval $[ s, t]$:

$$p _ {ij} ( s, t) = {\mathsf P} \{ \xi ( t) = j \mid \xi ( s) = i \} ,\ s< t.$$

In view of the basic property of a Markov chain, for any states $i, j \in S$ (where $S$ is the set of all states of the chain) and any $s < t < u$,

$$p _ {ij} ( s, u) = \sum _ {k \in S } p _ {ik} ( s, t) p _ {kj} ( t, u).$$

One usually considers homogeneous Markov chains, for which the transition probabilities $p _ {ij} ( s, t)$ depend on the length of $[ s, t]$ but not on its position on the time axis:

$$p _ {ij} ( s, t) = p _ {ij} ( t- s).$$

For any states $i$ and $j$ of a homogeneous Markov chain with discrete time, the sequence $p _ {ij} ( n)$ has a Cesàro limit, i.e.

$$\lim\limits _ {n \rightarrow \infty } \frac{1}{n} \sum _ { k= 1} ^ { n } p _ {ij} ( k) \geq 0.$$

Subject to certain additional conditions (and also for chains with continuous time), the limit exists also in the usual sense. See Markov chain, ergodic; Markov chain, class of positive states of a.

The transition probabilities $p _ {ij} ( t)$ for a Markov chain with discrete time are determined by the values of $p _ {ij} ( 1)$, $i, j \in S$; for any $t > 0$, $i \in S$,

$$\sum _ {j \in S } p _ {ij} ( t) = 1.$$

In the case of Markov chains with continuous time it is usually assumed that the transition probabilities satisfy the following additional conditions: All the $p _ {ij} ( t)$ are measurable as functions of $t \in ( 0, \infty )$,

$$\lim\limits _ {t \downarrow 0 } p _ {ij} ( t) = 0 \ \ ( i \neq j),\ \ \lim\limits _ {t \downarrow 0 } p _ {ii} ( t) = 1 ,\ \ i, j \in S.$$

Under these assumptions the following transition rates exist:

$$\tag{1 } \lambda _ {ij} = \lim\limits _ {t \downarrow 0 } \frac{1}{t} ( p _ {ij} ( t) - p _ {ij} ( 0)) \leq \infty ,\ \ i, j \in S;$$

if all the $\lambda _ {ij}$ are finite and if $\sum _ {j \in S } \lambda _ {ij} = 0$, $i \in S$, then the $p _ {ij} ( t)$ satisfy the Kolmogorov–Chapman system of differential equations

$$\tag{2 } p _ {ij} ^ \prime ( t) = \sum _ {k \in S } \lambda _ {ik} p _ {kj} ( t),\ \ p _ {ij} ^ \prime ( t) = \sum _ {k \in S } \lambda _ {kj} p _ {ik} ( t)$$

with the initial conditions $p _ {ii} ( 0) = 1$, $p _ {ij} ( 0) = 0$, $i \neq j$, $i, j \in S$ (see also Kolmogorov equation; Kolmogorov–Chapman equation).

If a Markov chain is specified by means of the transition rates (1), then the transition probabilities $p _ {ij} ( t)$ satisfy the conditions

$$p _ {ij} ( t) \geq 0,\ \ \sum _ {j \in S } p _ {ij} ( t) \leq 1,\ \ i, j \in S,\ \ t > 0;$$

chains for which $\sum _ {j \in S } p _ {ij} ( t) < 1$ for certain $i \in S$ and $t > 0$ are called defective (in this case the solution to (2) is not unique); if $\sum _ {j \in S } p _ {ij} ( t) = 1$ for all $i \in S$ and $t > 0$, the chain is called proper.

Example. The Markov chain $\xi ( t)$ with set of states $\{ 0, 1 ,\dots \}$ and transition densities

$$\lambda _ {i,i+1} = - \lambda _ {ii} = \lambda _ {i} > 0,\ \ \lambda _ {ij} = 0 \ \ ( i \neq j \neq i+ 1)$$

(i.e., a pure birth process) is defective if and only if

$$\sum _ { i= 0} ^ \infty \frac{1}{\lambda _ {i} } < \infty .$$

Let

$$\tau _ {0n} = \inf \{ {t > 0 } : {\xi ( t) = n ( \xi ( 0) = 0) } \} ,$$

$$\tau = \lim\limits _ {n \rightarrow \infty } \tau _ {0n} ;$$

then

$${\mathsf E} \tau = \sum _ { i= 1} ^ \infty \frac{1}{\lambda _ {i} }$$

and for ${\mathsf E} \tau < \infty$ one has ${\mathsf P} \{ \tau < \infty \} = 1$, i.e. the path of $\xi ( t)$" tends to infinity in a finite time with probability 1" (see also Branching processes, regularity of).

References

[1]	K.L. Chung, "Markov chains with stationary probability densities" , Springer (1967)

Comments

For additional references see also Markov chain; Markov process.

In (1), $\lambda _ {ij} \geq 0$ if $i \neq j$ and $\lambda _ {ii} \leq 0$.

References

[a1]	M. Iosifescu, "Finite Markov processes and their applications" , Wiley (1980)
[a2]	D. Revuz, "Markov chains" , North-Holland (1984)