View source for Bernstein-von Mises theorem

Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103701.png" /> be independent identically distributed random variables with a probability density depending on a parameter <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103702.png" /> (cf. [[Random variable|Random variable]]; [[Probability distribution|Probability distribution]]). Suppose that an a priori distribution for <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103703.png" /> is chosen. One of the fundamental theorems in the asymptotic theory of Bayesian inference (cf. [[Bayesian approach|Bayesian approach]]) is concerned with the convergence of the a posteriori density of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103704.png" />, given <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103705.png" />, to the normal density. In other words, the a posteriori distribution tends to look like a [[Normal distribution|normal distribution]] asymptotically. This phenomenon was first noted in the case of independent and identically distributed observations by P.S. Laplace. A related, but different, result was proved by S.N. Bernstein [[#References|[a2]]], who considered the a posteriori distribution of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103706.png" /> given the average <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103707.png" />. R. von Mises [[#References|[a12]]] extended the result to a posteriori distributions conditioned by a finite number of differentiable functionals of the empirical distribution function. L. Le Cam [[#References|[a5]]] studied the problem in his work on asymptotic properties of maximum likelihood and related Bayesian estimates. The Bernstein–von Mises theorem about convergence in the <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103708.png" />-mean for the case of independent and identically distributed random variables reads as follows, see [[#References|[a3]]].

Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b1103709.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037010.png" />, be independent identically distributed random variables with probability density <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037011.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037012.png" />. Suppose <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037013.png" /> is open and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037014.png" /> is an a priori probability density on <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037015.png" /> which is continuous and positive in an open neighbourhood of the true parameter <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037016.png" />. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037017.png" />. Suppose that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037018.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037019.png" /> exist and are continuous in <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037020.png" />. Further, suppose that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037021.png" /> is continuous, with <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037022.png" />. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037023.png" /> be a non-negative function satisfying

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037024.png" /></td> </tr></table>

for some <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037025.png" />. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037026.png" /> be a maximum-likelihood estimator of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037027.png" /> based on <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037028.png" /> (cf. [[Maximum-likelihood method|Maximum-likelihood method]]) and let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037029.png" /> be the corresponding likelihood function. It is known that under certain regularity conditions there exists a compact neighbourhood <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037030.png" /> of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037031.png" /> such that:

<img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037032.png" /> almost surely;

<img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037033.png" /> for large <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037034.png" />;

<img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037035.png" /> converges in distribution (cf. [[Convergence in distribution|Convergence in distribution]]) to the normal distribution with mean <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037036.png" /> and variance <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037037.png" /> as <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037038.png" />.

Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037039.png" /> denote the a posteriori density of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037040.png" /> given the observation <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037041.png" /> and the a priori probability density <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037042.png" />, that is,

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037043.png" /></td> </tr></table>

Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037044.png" />. Then <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037045.png" /> is the a posteriori density of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037046.png" />.

A generalized version of the Bernstein–von Mises theorem, under the assumptions stated above and some addition technical conditions, is as follows.

If, for every <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037047.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037048.png" />,

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037049.png" /></td> </tr></table>

then

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037050.png" /></td> </tr></table>

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037051.png" /></td> </tr></table>

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037052.png" /></td> </tr></table>

For <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037053.png" /> one finds that the a posteriori density converges to the normal density in <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/b/b110/b110370/b11037054.png" />-mean convergence. The result can be extended to a multi-dimensional parameter. As an application of the above theorem, it can be shown that the Bayesian estimator is strongly consistent and asymptotically efficient for a suitable class of loss functions (cf. [[#References|[a11]]]). For rates of convergence see [[#References|[a4]]], [[#References|[a7]]], [[#References|[a8]]].

B.L.S. Prakasa Rao [[#References|[a6]]] has generalized the result to arbitrary discrete-time stochastic processes (cf. [[#References|[a1]]]); for extensions to diffusion processes and diffusion fields, see [[#References|[a9]]], [[#References|[a10]]].

====References====
<table><TR><TD valign="top">[a1]</TD> <TD valign="top">  I.V. Basawa,   B.L.S. Prakasa Rao,   "Statistical inference for stochastic processes" , Acad. Press  (1980)</TD></TR><TR><TD valign="top">[a2]</TD> <TD valign="top">  S.N. Bernstein,   "Theory of probability"  (1917)  (In Russian)</TD></TR><TR><TD valign="top">[a3]</TD> <TD valign="top">  J.D. Borwanker,   G. Kallianpur,   B.L.S. Prakasa Rao,   "The Bernstein–von Mises theorem for Markov processes"  ''Ann. Math. Stat.'' , '''43'''  (1971)  pp. 1241–1253</TD></TR><TR><TD valign="top">[a4]</TD> <TD valign="top">  C. Hipp,   R. Michael,   "On the Bernstein–von Mises approximation of posterior distribution"  ''Ann. Stat.'' , '''4'''  (1976)  pp. 972–980</TD></TR><TR><TD valign="top">[a5]</TD> <TD valign="top">  L. Le Cam,   "On some asymptotic properties of maximum likelihood estimates and related Bayes estimates"  ''Univ. California Publ. Stat.'' , '''1'''  (1953)  pp. 277–330</TD></TR><TR><TD valign="top">[a6]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Statistical inference for stochastic processes"  G. Sankaranarayanan (ed.) , ''Proc. Advanced Symp. on Probability and its Applications'' , Annamalai Univ.  (1976)  pp. 43–150</TD></TR><TR><TD valign="top">[a7]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Rate of convergence of Bernstein–von Mises approximation for Markov processes"  ''Serdica'' , '''4'''  (1978)  pp. 36–42</TD></TR><TR><TD valign="top">[a8]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "The equivalence between (modified) Bayes estimator and maximum likelihood estimator for Markov processes"  ''Ann. Inst. Statist. Math.'' , '''31'''  (1979)  pp. 499–513</TD></TR><TR><TD valign="top">[a9]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "The Bernstein–von Mises theorem for a class of diffusion processes"  ''Teor. Sluch. Prots.'' , '''9'''  (1981)  pp. 95–104  (In Russian)</TD></TR><TR><TD valign="top">[a10]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "On Bayes estimation for diffusion fields"  J.K. Ghosh (ed.)  J. Roy (ed.) , ''Statistics: Applications and New Directions'' , Statistical Publishing Soc.  (1984)  pp. 504–511</TD></TR><TR><TD valign="top">[a11]</TD> <TD valign="top">  B.L.S. Prakasa Rao,   "Asymptotic theory of statistical inference" , Wiley  (1987)</TD></TR><TR><TD valign="top">[a12]</TD> <TD valign="top">  R. von Mises,   "Wahrscheinlichkeitsrechnung" , Springer  (1931)</TD></TR></table>