Lyapunov stochastic function

A non-negative function $ V ( t , x ) $ for which the pair $ ( V ( t , X ( t) ) , F _ {t} ) $ is a supermartingale for some random process $ X $ up to the instant $ t $( cf. also Martingale). If $ X ( t) $ is a Markov process, then the Lyapunov stochastic function is a function for which the Lyapunov stochastic operator

$$ LV {( t ,x ) } = $$

$$ = \ \lim\limits _ {h \rightarrow 0 } \frac{1}{h} {\mathsf E} [ V ( t+ h ,\ X ( t+ h)) - V ( t, X ( t)) \mid X ( t) = x ] $$

is non-positive. The operator $ L $ is the infinitesimal operator of the process $ ( t , X ( t)) $, and so the verification of the condition $ LV \leq 0 $ is easily carried out in specific cases. The operator $ L $ goes into the usual Lyapunov operator $ dV ( t , X ( t)) / dt $ when the process $ X $ is determinate and is described by a system of differential equations. By means of the Lyapunov stochastic function it is possible to verify a number of qualitative properties of the trajectories of $ X ( t) $; their role in the theory of random processes is similar to the role of the classical Lyapunov function in the theory of systems of differential equations.

Functions $ V ( t , x ) $ for which $ ( V ( t , X ( t)) , F _ {t} ) $ is not a supermartingale, but from which one can readily form a supermartingale, are sometimes also called Lyapunov stochastic functions. Below typical results are presented on the qualitative behaviour of trajectories of Markov processes in terms of a Lyapunov stochastic function.

1) If $ X ( t) $ is a right-continuous strong Markov process in $ \mathbf R ^ {k} $, defined up to the instant $ \tau $ of first leaving an arbitrary compact set, and if there is a Lyapunov stochastic function $ V ( t , x ) $, $ t > 0 $, $ x \in \mathbf R ^ {k} $, and a constant $ c $ such that

$$ \inf _ {t , | x| > R } V ( t , x ) \rightarrow \infty \ \ \textrm{ as } R \rightarrow \infty ,\ \ L V \leq c V , $$

then

$$ {\mathsf P} \{ \tau < \infty \mid X ( 0) = x \} = 1 $$

for any $ x \in \mathbf R ^ {k} $; that is, the process $ X $ is defined for all $ t > 0 $( is indefinitely extendable).

2) For the stationary Markov process in $ \mathbf R ^ {k} $ corresponding to a transition function $ P ( t , x , A ) $ to exist it is sufficient that there should be a function $ V ( x) \geq 0 $ for which

$$ \sup _ {| x| > R } L V ( x) \rightarrow - \infty $$

as $ R \rightarrow \infty $.

By means of the Lyapunov stochastic function one can carry over to Markov processes the main theorems of the direct Lyapunov method; these functions have also found application in the investigation of processes in discrete time.

References

[1]	H.J. Kushner, "Stochastic stability and control" , Acad. Press (1967)
[2]	R.Z. [R.Z. Khas'minskii] Has'minskii, "Stochastic stability of differential equations" , Sijthoff & Noordhoff (1980) (Translated from Russian)
[3]	V.V. Kalashnikov, "Qualitative analysis of the behaviour of complex systems by the method of test functions" , Moscow (1978) (In Russian)

Comments

The phrase stochastic Lyapunov function is more common than "Lyapunov stochastic function" .

Recently, stochastic Lyapunov functions have been used to prove convergence of recursive algorithms driven by stochastic processes. Convergence problems of this type arise in system identification and adaptive control.

References

[a1]	G.C. Goodwin, P.J. Ramagadge, P.E. Caines, "Discrete time stochastic adaptive control" SIAM J. Control Optim. , 19 (1981) pp. 829–853
[a2]	M. Metivier, P. Priouret, "Applications of a Kushner and Clark lemma to general classes of stochastic algorithms" IEEE Trans. Inform. Theory , 30 (1984) pp. 140–151
[a3]	V. Solo, "The convergence of AML" IEEE Trans. Autom. Control , 24 (1979) pp. 958–962
[a4]	J.H. van Schuppen, "Convergence results for continuous-time stochastic filtering algorithms" J. Math. Anal. Appl. , 96 (1983) pp. 209–225