Semi-Markov process

A stochastic process $ X ( t) $ with a finite or countable set of states $ N = \{ 1 , 2 , . . . \} $, having stepwise trajectories with jumps at times $ 0 < \tau _ {1} < \tau _ {2} < \dots $ and such that the values $ X ( \tau _ {n} ) $ at its jump times form a Markov chain with transition probabilities

$$ p _ {ij} = {\mathsf P} \{ X ( \tau _ {n} ) = j \mid X ( \tau _ {n-} 1 ) = i \} . $$

The distributions of the jump times $ \tau _ {n} $ are described in terms of the distribution functions $ F _ {ij} ( x) $ as follows:

$$ {\mathsf P} \{ \tau _ {n} - \tau _ {n-} 1 \leq x ,\ X ( \tau _ {n} ) = j \mid X ( \tau _ {n-} 1 ) = i \} = p _ {ij} F _ {ij} ( x) $$

(and, moreover, they are independent of the states of the process at earlier moments of time). If

$$ F _ {ij} ^ { \prime } ( x) = e ^ {- \lambda _ {ij} x } ,\ \ x \geq 0 , $$

for all $ i , j \in N $, then the semi-Markov process $ X ( t) $ is a continuous-time Markov chain. If all the distributions degenerate to a point, the result is a discrete-time Markov chain.

Semi-Markov processes provide a model for many processes in queueing theory and reliability theory. Related to semi-Markov processes are Markov renewal processes (see Renewal theory), which describe the number of times the process $ X ( t) $ is in state $ i \in N $ during the time $ [ 0 , t ] $.

In analytic terms, the investigation of semi-Markov processes and Markov renewal processes reduces to a system of integral equations — the renewal equations.

References

[1]	V.S. Korolyuk, A.F. Turbin, "Semi-Markov processes and their applications" , Kiev (1976) (In Russian)

References

[a1]	E. Cinlar, "Introduction to stochastic processes" , Prentice-Hall (1975) pp. Chapt. 10