View source for Bernstein-von Mises theorem

<!--
b1103701.png
$#A+1 = 54 n = 0
$#C+1 = 54 : ~/encyclopedia/old_files/data/B110/B.1100370 Bernstein\ANDvon Mises theorem
Automatically converted into TeX, above some diagnostics.
Please remove this comment and the {{TEX|auto}} line below,
if TeX found to be correct.
-->

{{TEX|auto}}
{{TEX|done}}

Let  $  \{ {X _ {j} } : {j \geq  1 } \} $
be independent identically distributed random variables with a probability density depending on a parameter  $  \theta $(
cf. [[Random variable|Random variable]]; [[Probability distribution|Probability distribution]]). Suppose that an a priori distribution for  $  \theta $
is chosen. One of the fundamental theorems in the asymptotic theory of Bayesian inference (cf. [[Bayesian approach|Bayesian approach]]) is concerned with the convergence of the a posteriori density of  $  \theta $, 
given  $  X _ {1} \dots X _ {n} $, 
to the normal density. In other words, the a posteriori distribution tends to look like a [[Normal distribution|normal distribution]] asymptotically. This phenomenon was first noted in the case of independent and identically distributed observations by P.S. Laplace. A related, but different, result was proved by S.N. Bernstein [[#References|[a2]]], who considered the a posteriori distribution of  $  \theta $
given the average  $  n ^ {-1 } ( X _ {1} + \dots + X _ {n} ) $. 
R. von Mises [[#References|[a12]]] extended the result to a posteriori distributions conditioned by a finite number of differentiable functionals of the empirical distribution function. L. Le Cam [[#References|[a5]]] studied the problem in his work on asymptotic properties of maximum likelihood and related Bayesian estimates. The Bernstein–von Mises theorem about convergence in the  $  L _ {1} $-
mean for the case of independent and identically distributed random variables reads as follows, see [[#References|[a3]]].

Let  $  X _ {i} $, 
$  1 \leq  i \leq  n $, 
be independent identically distributed random variables with probability density  $  f ( x, \theta ) $, 
$  \theta \in \Theta \subset  \mathbf R $. 
Suppose  $  \Theta $
is open and  $  \lambda $
is an a priori probability density on  $  \Theta $
which is continuous and positive in an open neighbourhood of the true parameter  $  \theta _ {0} $. 
Let  $  h ( x, \theta ) = { \mathop{\rm log} } f ( x, \theta ) $. 
Suppose that  $  { {\partial  h } / {\partial  \theta } } $
and  $  { {\partial   ^ {2} h } / {\partial  \theta  ^ {2} } } $
exist and are continuous in  $  \theta $. 
Further, suppose that  $  i ( \theta ) = - {\mathsf E} _  \theta  [ { {\partial   ^ {2} h } / {\partial  \theta  ^ {2} } } ] $
is continuous, with  $  0 < i ( \theta ) < \infty $. 
Let  $  K ( \cdot ) $
be a non-negative function satisfying

$$ 
\int\limits _ {- \infty } ^  \infty   {K ( t ) { \mathop{\rm exp} } \left [ - {
\frac{( i ( \theta _ {0} ) - \epsilon ) t  ^ {2} }{2}
 } \right ] }  {d t } < \infty
$$

for some  $  0 < \epsilon < i ( \theta _ {0} ) $. 
Let  $  {\widehat \theta   } _ {n} $
be a maximum-likelihood estimator of  $  \theta $
based on  $  X _ {1} \dots X _ {n} $(
cf. [[Maximum-likelihood method|Maximum-likelihood method]]) and let  $  L _ {n} ( \theta ) $
be the corresponding likelihood function. It is known that under certain regularity conditions there exists a compact neighbourhood  $  U _ {\theta _ {0}  } $
of  $  \theta _ {0} $
such that:

$  {\widehat \theta   } _ {n} \rightarrow \theta _ {0} $
almost surely;

$  ( { {\partial  { \mathop{\rm log} } L _ {n} ( \theta ) } / {\partial  \theta } } ) \mid  _ {\theta = {\widehat \theta   }  _ {n} } = 0 $
for large  $  n $;

$  n ^ {1/2 } ( {\widehat \theta   } _ {n} - \theta _ {0} ) $
converges in distribution (cf. [[Convergence in distribution|Convergence in distribution]]) to the normal distribution with mean  $  0 $
and variance  $  {1 / {i ( \theta _ {0} ) } } $
as  $  n \rightarrow \infty $.

Let  $  f _ {n} ( \theta \mid  x _ {1} \dots x _ {n} ) $
denote the a posteriori density of  $  \theta $
given the observation  $  ( x _ {1} \dots x _ {n} ) $
and the a priori probability density  $  \lambda ( \theta ) $, 
that is,

$$ 
f _ {n} ( \theta \mid  x _ {1} \dots x _ {n} ) = {
\frac{\prod _ {i = 1 } ^ { n }  f ( x _ {i} , \theta ) \lambda ( \theta ) }{\int\limits _  \Theta  {\prod _ {i = 1 } ^ { n }  f ( x _ {i} , \phi ) \lambda ( \phi ) }  {d \phi } }
 } .
$$

Let  $  f _ {n}  ^ {*} ( t \mid  x _ {1} \dots x _ {n} ) = n ^ {- 1/2 } f _ {n} ( {\widehat \theta   } _ {n} + tn ^ {- 1/2 } ) $. 
Then  $  f _ {n}  ^ {*} ( t \mid  x _ {1} \dots x _ {n} ) $
is the a posteriori density of  $  t = n ^ {1/2 } ( \theta - {\widehat \theta   } _ {n} ) $.

A generalized version of the Bernstein–von Mises theorem, under the assumptions stated above and some addition technical conditions, is as follows.

If, for every  $  h > 0 $
and  $  \delta > 0 $,

$$ 
e ^ {- n \delta } \int\limits _ {\left | t \right | > h } {K ( n ^ {1/2 } t ) \lambda ( {\widehat \theta   } _ {n} + t ) }  {d t } \rightarrow 0  \textrm{ a.s.  }  [ {\mathsf P} _ {\theta _ {0}  } ] ,
$$

then

$$ 
{\lim\limits } _ {n \rightarrow \infty } \int\limits _ {- \infty } ^  \infty   {K ( t ) } \cdot
$$

$$ 
\cdot 
{\left | {f _ {n}  ^ {*} ( t \mid  X _ {1} \dots X _ {n} ) - \left ( {
\frac{i ( \theta _ {0} ) }{2 \pi }
 } \right ) ^ { {1 / 2 } } e ^ {- {
\frac{1}{2}
 } i ( \theta _ {0} ) t  ^ {2} } } \right | }  {d t }  =
$$

$$ 
= 
0  \textrm{ a.s.  }  [ {\mathsf P} _ {\theta _ {0}  } ] .
$$

For  $  K ( t ) \equiv 1 $
one finds that the a posteriori density converges to the normal density in  $  L _ {1} $-
mean convergence. The result can be extended to a multi-dimensional parameter. As an application of the above theorem, it can be shown that the Bayesian estimator is strongly consistent and asymptotically efficient for a suitable class of loss functions (cf. [[#References|[a11]]]). For rates of convergence see [[#References|[a4]]], [[#References|[a7]]], [[#References|[a8]]].

B.L.S. Prakasa Rao [[#References|[a6]]] has generalized the result to arbitrary discrete-time stochastic processes (cf. [[#References|[a1]]]); for extensions to diffusion processes and diffusion fields, see [[#References|[a9]]], [[#References|[a10]]].

====References====
<table><TR><TD valign="top">[a1]</TD> <TD valign="top">  I.V. Basawa,   B.L.S. Prakasa Rao,   "Statistical inference for stochastic processes" , Acad. Press  (1980)</TD></TR><TR><TD valign="top">[a2]</TD> <TD valign="top">  S.N. Bernstein,   "Theory of probability"  (1917)  (In Russian)</TD></TR><TR><TD valign="top">[a3]</TD> <TD valign="top">  J.D. Borwanker,   G. Kallianpur,   B.L.S. Prakasa Rao,   "The Bernstein–von Mises theorem for Markov processes"  ''Ann. Math. Stat.'' , '''43'''  (1971)  pp. 1241–1253</TD></TR><TR><TD valign="top">[a4]</TD> <TD valign="top">  C. Hipp,   R. Michael,   "On the Bernstein–von Mises approximation of posterior distribution"  ''Ann. Stat.'' , '''4'''  (1976)  pp. 972–980</TD></TR><TR><TD valign="top">[a5]</TD> <TD valign="top">  L. Le Cam,   "On some asymptotic properties of maximum likelihood estimates and related Bayes estimates"  ''Univ. California Publ. Stat.'' , '''1'''  (1953)  pp. 277–330</TD></TR><TR><TD valign="top">[a6]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Statistical inference for stochastic processes"  G. Sankaranarayanan (ed.) , ''Proc. Advanced Symp. on Probability and its Applications'' , Annamalai Univ.  (1976)  pp. 43–150</TD></TR><TR><TD valign="top">[a7]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Rate of convergence of Bernstein–von Mises approximation for Markov processes"  ''Serdica'' , '''4'''  (1978)  pp. 36–42</TD></TR><TR><TD valign="top">[a8]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "The equivalence between (modified) Bayes estimator and maximum likelihood estimator for Markov processes"  ''Ann. Inst. Statist. Math.'' , '''31'''  (1979)  pp. 499–513</TD></TR><TR><TD valign="top">[a9]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "The Bernstein–von Mises theorem for a class of diffusion processes"  ''Teor. Sluch. Prots.'' , '''9'''  (1981)  pp. 95–104  (In Russian)</TD></TR><TR><TD valign="top">[a10]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "On Bayes estimation for diffusion fields"  J.K. Ghosh (ed.)  J. Roy (ed.) , ''Statistics: Applications and New Directions'' , Statistical Publishing Soc.  (1984)  pp. 504–511</TD></TR><TR><TD valign="top">[a11]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Asymptotic theory of statistical inference" , Wiley  (1987)</TD></TR><TR><TD valign="top">[a12]</TD> <TD valign="top">  R. von Mises,   "Wahrscheinlichkeitsrechnung" , Springer  (1931)</TD></TR></table>