Markov process

process without after-effects

2020 Mathematics Subject Classification: Primary: 60Jxx [MSN][ZBL]

A stochastic process whose evolution after a given time does not depend on the evolution before , given that the value of the process at is fixed (briefly; the "future" and "past" of the process are independent of each other for a known "present" ).

The defining property of a Markov process is commonly called the Markov property; it was first stated by A.A. Markov . However, in the work of L. Bachelier it is already possible to find an attempt to discuss Brownian motion as a Markov process, an attempt which received justification later in the research of N. Wiener (1923). The basis of the general theory of continuous-time Markov processes was laid by A.N. Kolmogorov .

The Markov property.

There are essentially distinct definitions of a Markov process. One of the more widely used is the following. On a probability space let there be given a stochastic process , , taking values in a measurable space , where is a subset of the real line . Let (respectively, ) be the -algebra in generated by the variables for (), where . In other words, (respectively, ) is the collection of events connected with the evolution of the process up to time (starting from time) . is called a Markov process if (almost certainly) for all , the Markov property

(1)

holds, or, what is the same, if for any and ,

(2)

A Markov process for which is contained in the natural numbers is called a Markov chain (however, the latter term is mostly associated with the case of an at most countable ). If is an interval in and is at most countable, a Markov process is called a continuous-time Markov chain. Examples of continuous-time Markov processes are furnished by diffusion processes (cf. Diffusion process) and processes with independent increments (cf. Stochastic process with independent increments), including Poisson and Wiener processes (cf. Poisson process; Wiener process).

In what follows the discussion will concern only the case , for the sake of being specific. The formulas (1) and (2) give an explicit interpretation of the principle of independence of "past" and "future" events when the "present" is known, but the definition of Markov process based on them has proved to be insufficiently flexible in the numerous situations where one is obliged to consider not one, but a collection of conditions of the type (1) or (2) corresponding to different, but in some sense consistent, measures . Such reasoning has led to the acceptance of the following definitions (see , ).

Suppose one is given:

a) a measurable space , where the -algebra contains all one-point sets in ;

b) a measurable space , equipped with a family of -algebras , , such that if ;

c) a function ( "trajectory" ) , defining for and a measurable mapping from to ;

d) for each and a probability measure on the -algebra such that the function is measurable with respect to , if and .

The collection is called a (non-terminating) Markov process given on if -almost certainly

(3)

for any and . Here is the space of elementary events, is the phase space or state space and is the transition function or transition probability of . If is endowed with a topology and is the collection of Borel sets in , then it is commonly said that the Markov process is given on . Usually included in the definition of a Markov process is the requirement that , and then , , is interpreted as the probability of under the condition that .

The following question arises: Is every Markov transition function , given on a measurable space , the transition function of some Markov process? The answer is affirmative if, for example, is a separable, locally compact space and is the family of Borel sets in . In addition, let be a complete metric space and let

for any , where

and is the complement of the -neighbourhood of . Then the corresponding Markov process can be taken to be right-continuous and having left limits (that is, its trajectories can be chosen so). The existence of a continuous Markov process is guaranteed by the condition as (see , ).

In the theory of Markov processes most attention is given to homogeneous (in time) processes. The corresponding definition assumes one is given a system of objects a)–d) with the difference that the parameters and may now only take the value 0. Even the notation can be simplified:

Subsequently, homogeneity of is assumed. That is, it is required that for any and there is an such that for . Because of this, on the -algebra , the smallest -algebra in containing the events , the time shift operators are defined, which preserve the operations of union, intersection and difference of sets, and for which

where , .

The collection is called a (non-terminating) homogeneous Markov process given on if -almost certainly

(4)

for , and . The transition function of is taken to be , where, unless otherwise indicated, it is required that . It is useful to bear in mind that in the verification of (4) it is only necessary to consider sets of the form , where , , and in (4), may always replaced by the -algebra equal to the intersection of the completions of relative to all possible measures . Often, one fixes on a probability measure (the "initial distribution" ) and considers a random Markov function , where is the measure on given by

A Markov process is called progressively measurable if for each the function induces a measurable mapping from to , where is the -algebra of Borel subsets of . A right-continuous Markov process is progressively measurable. There is a method for reducing the non-homogeneous case to the homogeneous case (see ), and in what follows homogeneous Markov processes will be discussed.

The strong Markov property.

Suppose that, on a measurable space , a Markov process is given. A function is called a Markov moment (stopping time) if for . Here a set is considered in the family if for (most often is interpreted as the family of events connected with the evolution of up to time ). For , set

A progressively-measurable Markov process is called a strong Markov process if for any Markov moment and all , and , the relation

(5)

(the strong Markov property) is satisfied -almost certainly in . In the verification of (5) it suffices to consider only sets of the form where , ; in this case . For example, any right-continuous Feller–Markov process on a topological space is a strong Markov process. A Markov process is called a Feller–Markov process if the function

is continuous whenever is continuous and bounded.

In the case of strong Markov processes various subclasses have been distinguished. Let the Markov transition function , given on a locally compact metric space , be stochastically continuous:

for any neighbourhood of each point . If maps the class of continuous functions that vanish at infinity into itself, then corresponds to a standard Markov process . That is, a right-continuous strong Markov process for which: 1) for and for ; 2) , -almost certainly on the set , where and () are Markov moments that are non-decreasing as increases.

Terminating Markov processes.

Frequently, a physical system can be best described using a non-terminating Markov process, but only in a time interval of random length. In addition, even simple transformations of a Markov process may lead to processes with trajectories given on random intervals (see Functional of a Markov process). Guided by these considerations one introduces the notion of a terminating Markov process.

Let be a homogeneous Markov process in a phase space , having a transition function , and let there be a point and a function such that for and otherwise (unless stated otherwise, take ). A new trajectory is given for by the equality , and is defined as the trace of on the set .

The collection , where , is called the terminating Markov process obtained from by censoring (or killing) at the time . The variable is called the censoring time or lifetime of the terminating Markov process. The phase space of the new process is , where is the trace of the -algebra in . The transition function of a terminating Markov process is the restriction of to the set , , . The process is called a strong Markov process or a standard Markov process if has the corresponding property. A non-terminating Markov process can be considered as a terminating Markov process with censoring time . A non-homogeneous terminating Markov process is defined similarly.

M.G. Shur

Markov processes and differential equations.

A Markov process of Brownian-motion type is closely connected with partial differential equations of parabolic type. The transition density of a diffusion process satisfies, under certain additional assumptions, the backward and forward Kolmogorov equations (cf. Kolmogorov equation):

(6)

(7)

The function is the Green's function of the equations (6)–(7), and the first known methods for constructing diffusion processes were based on existence theorems for this function for the partial differential equations (6)–(7). For a time-homogeneous process the operator coincides on smooth functions with the infinitesimal operator of the Markov process (see Transition-operator semi-group).

The expectations of various functionals of diffusion processes are solutions of boundary value problems for the differential equation . Let be the expectation with respect to the measure . Then the function satisfies (6) for and .

Similarly, the function

satisfies, for ,

and .

Let be the time at which the trajectories of first hit the boundary of a domain , and let . Then, under certain conditions, the function

satisfies

and takes the value on the set

The solution of the first boundary value problem for a general second-order linear parabolic equation

(8)

can, under fairly general assumptions, be described in the form

(9)

When the operator and the functions and do not depend on , a representation similar to (9) is possible also for the solution of a linear elliptic equation. More precisely, the function

(10)

is, under certain assumptions, the solution of

(11)

When is degenerate or is not sufficiently "smooth" , the boundary values need not be taken by the functions (9), (10) at individual points or on whole sets. The notion of a regular boundary point for has a probabilistic interpretation. At regular points the boundary values are attained by (9), (10). The solution of (8) and (11) allows one to study the properties of the corresponding diffusion processes and functionals of them.

There are methods for constructing Markov processes which do not rely on the construction of solutions of (6) and (7). For example, the method of stochastic differential equations (cf. Stochastic differential equation), of absolutely-continuous change of measure, etc. This situation, together with the formulas (9) and (10), gives a probabilistic route to the construction and study of the properties of boundary value problems for (8) and also to the study of properties of the solutions of the corresponding elliptic equation.

Since the solution of a stochastic differential equation is insensitive to degeneracy of , probabilistic methods can be applied to construct solutions of degenerate elliptic and parabolic differential equations. The extension of the averaging principle of N.M. Krylov and N.N. Bogolyubov to stochastic differential equations allows one, with the help of (9), to obtain corresponding results for elliptic and parabolic differential equations. It turns out that certain difficult problems in the investigation of properties of solutions of equations of this type with small parameters in front of the highest derivatives can be solved by probabilistic arguments. Even the solution of the second boundary value problem for (6) has a probabilistic meaning. The formulation of boundary value problems for unbounded domains is closely connected with recurrence in the corresponding diffusion process.

In the case of a time-homogeneous process ( is independent of ), a positive solution of coincides, under certain assumptions and up to a multiplicative constant, with the stationary density of the distribution of a Markov chain. Probabilistic arguments turn out to be useful even for boundary value problems for non-linear parabolic equations.

R.Z. Khas'minskii

References to all sections are given below.

References

[M]	A.A. Markov, "?", Izv. Fiz.-Mat. Obshch. Kazan. Univ. , 15 : 4 (1906) pp. 135–156
[B]	L. Bachelier, "?", Ann. Sci. Ecole Norm. Sup. , 17 (1900) pp. 21–86
[Ko]	A.N. Kolmogorov, "Ueber die analytischen Methoden in der Wahrscheinlichkeitsrechnung" Math. Ann. , 104 (1931) pp. 415–458
[C]	K.L. Chung, "Markov chains with stationary transition probabilities" , Springer (1967) MR0217872 Zbl 0146.38401
[F]	W. Feller, "The general diffusion operator and positivity-preserving semi-groups in one dimension" Ann. of Math. , 60 (1954) pp. 417–436 MR0065809
[DY]	E.B. Dynkin, A.A. Yushkevich, "Strong Markov processs" Theor. Veroyatnost. i Primenen. , 1 : 1 (1956) pp. 149–155 (In Russian) (English abstract)
[H]	G.A. Hunt, "Markov processes and potentials I" Illinois J. Math. , 1 (1957) pp. 44–93
[H2]	G.A. Hunt, "Markov processes and potentials II" Illinois J. Math. , 1 (1957) pp. 316–369
[H3]	G.A. Hunt, "Markov processes and potentials III" Illinois J. Math. , 2 (1958) pp. 151–213
[De]	C. Dellacherie, "Capacités et processus stochastiques" , Springer (1972) MR0448504 Zbl 0246.60032
[Dy]	E.B. Dynkin, "Theory of Markov processes" , Pergamon (1960) (Translated from Russian) MR2305744 MR1009436 MR0245096 MR1531923 MR0131900 MR0131898 Zbl 1116.60001 Zbl 0689.60003 Zbl 0222.60048 Zbl 0091.13605
[D2]	E.B. Dynkin, "Markov processes" , 1 , Springer (1965) (Translated from Russian) MR0193671 Zbl 0132.37901
[GS]	I.I. Gihman, A.V. Skorohod, "The theory of stochastic processes" , 2 , Springer (1979) (Translated from Russian) MR0651014 MR0651015 Zbl 0404.60061
[Fr]	M.I. Freidlin, "Markov processes and differential equations" Itogi Nauk. Teor. Veroyatnost. Mat. Statist. Teoret. Kibernet. 1966 (1967) pp. 7–58 (In Russian) MR0235308 Zbl 0863.60049
[Kh]	R.Z. Khas'minskii, "Principle of averaging for parabolic and elliptic partial differential equations and for Markov processes with small diffusion" Theor. Probab. Appl. , 8 (1963) pp. 1–21 Teor. Veroyatnost. i Primenen. , 8 : 1 (1963) pp. 3–25
[VF]	A.D. Venttsel', M.I. Freidlin, "Random perturbations of dynamical systems" , Springer (1984) (Translated from Russian) MR722136
[BG]	R.M. Blumenthal, R.K. Getoor, "Markov processes and potential theory" , Acad. Press (1968) MR0264757 Zbl 0169.49204
[G]	R.K. Getoor, "Markov processes: Ray processes and right processes" , Lect. notes in math. , 440 , Springer (1975) MR0405598 Zbl 0299.60051
[Kuz]	S.E. Kuznetsov, "Any Markov process in a Borel space has a transition function" Theor. Prob. Appl. , 25 (1980) pp. 384–388 Teor. Veroyatnost. i Primenen. , 25 : 2 (1980) pp. 389–393 MR0572574 Zbl 0456.60077 Zbl 0431.60071

References

[SV]	D.W. Stroock, S.R.S. Varadhan, "Multidimensional diffusion processes" , Springer (1979) MR0532498 Zbl 0426.60069
[C2]	K.L. Chung, "Lectures from Markov processes to Brownian motion" , Springer (1982) MR0648601 Zbl 0503.60073
[Do]	J.L. Doob, "Stochastic processes" , Wiley (1953) MR1570654 MR0058896 Zbl 0053.26802
[We]	A.D. Wentzell, "A course in the theory of stochastic processes" , McGraw-Hill (1981) (Translated from Russian) MR0781738 MR0614594 Zbl 0502.60001
[Kur]	T.G. Kurtz, "Markov processes" , Wiley (1986) MR0838085 Zbl 0592.60049
[F2]	W. Feller, "An introduction to probability theory and its applications" , 1–2 , Wiley (1966) MR0210154 Zbl 0138.10207
[Wa]	N. Wax (ed.), Selected papers on noise and stochastic processes , Dover, reprint (1954) Zbl 0059.11903
[Ka]	M. Kac, "Probability and related topics in physical sciences" , Interscience (1959) pp. Chapt. 4 MR0102849 Zbl 0087.33003
[Le]	P. Lévy, "Processus stochastiques et mouvement Brownien" , Gauthier-Villars (1965) MR0190953 Zbl 0137.11602
[Lo]	M. Loève, "Probability theory" , II , Springer (1978) MR0651017 MR0651018 Zbl 0385.60001