Bernstein-von Mises theorem

Let be independent identically distributed random variables with a probability density depending on a parameter (cf. Random variable; Probability distribution). Suppose that an a priori distribution for is chosen. One of the fundamental theorems in the asymptotic theory of Bayesian inference (cf. Bayesian approach) is concerned with the convergence of the a posteriori density of , given , to the normal density. In other words, the a posteriori distribution tends to look like a normal distribution asymptotically. This phenomenon was first noted in the case of independent and identically distributed observations by P.S. Laplace. A related, but different, result was proved by S.N. Bernstein [a2], who considered the a posteriori distribution of given the average . R. von Mises [a12] extended the result to a posteriori distributions conditioned by a finite number of differentiable functionals of the empirical distribution function. L. Le Cam [a5] studied the problem in his work on asymptotic properties of maximum likelihood and related Bayesian estimates. The Bernstein–von Mises theorem about convergence in the -mean for the case of independent and identically distributed random variables reads as follows, see [a3].

Let , , be independent identically distributed random variables with probability density , . Suppose is open and is an a priori probability density on which is continuous and positive in an open neighbourhood of the true parameter . Let . Suppose that and exist and are continuous in . Further, suppose that is continuous, with . Let be a non-negative function satisfying

for some . Let be a maximum-likelihood estimator of based on (cf. Maximum-likelihood method) and let be the corresponding likelihood function. It is known that under certain regularity conditions there exists a compact neighbourhood of such that:

almost surely;

for large ;

converges in distribution (cf. Convergence in distribution) to the normal distribution with mean and variance as .

Let denote the a posteriori density of given the observation and the a priori probability density , that is,

Let . Then is the a posteriori density of .

A generalized version of the Bernstein–von Mises theorem, under the assumptions stated above and some addition technical conditions, is as follows.

If, for every and ,

then

For one finds that the a posteriori density converges to the normal density in -mean convergence. The result can be extended to a multi-dimensional parameter. As an application of the above theorem, it can be shown that the Bayesian estimator is strongly consistent and asymptotically efficient for a suitable class of loss functions (cf. [a11]). For rates of convergence see [a4], [a7], [a8].

B.L.S. Prakasa Rao [a6] has generalized the result to arbitrary discrete-time stochastic processes (cf. [a1]); for extensions to diffusion processes and diffusion fields, see [a9], [a10].

References

[a1]	I.V. Basawa, B.L.S. Prakasa Rao, "Statistical inference for stochastic processes" , Acad. Press (1980)
[a2]	S.N. Bernstein, "Theory of probability" (1917) (In Russian)
[a3]	J.D. Borwanker, G. Kallianpur, B.L.S. Prakasa Rao, "The Bernstein–von Mises theorem for Markov processes" Ann. Math. Stat. , 43 (1971) pp. 1241–1253
[a4]	C. Hipp, R. Michael, "On the Bernstein–von Mises approximation of posterior distribution" Ann. Stat. , 4 (1976) pp. 972–980
[a5]	L. Le Cam, "On some asymptotic properties of maximum likelihood estimates and related Bayes estimates" Univ. California Publ. Stat. , 1 (1953) pp. 277–330
[a6]	B.L.S. Prakasa Rao, "Statistical inference for stochastic processes" G. Sankaranarayanan (ed.) , Proc. Advanced Symp. on Probability and its Applications , Annamalai Univ. (1976) pp. 43–150
[a7]	B.L.S. Prakasa Rao, "Rate of convergence of Bernstein–von Mises approximation for Markov processes" Serdica , 4 (1978) pp. 36–42
[a8]	B.L.S. Prakasa Rao, "The equivalence between (modified) Bayes estimator and maximum likelihood estimator for Markov processes" Ann. Inst. Statist. Math. , 31 (1979) pp. 499–513
[a9]	B.L.S. Prakasa Rao, "The Bernstein–von Mises theorem for a class of diffusion processes" Teor. Sluch. Prots. , 9 (1981) pp. 95–104 (In Russian)
[a10]	B.L.S. Prakasa Rao, "On Bayes estimation for diffusion fields" J.K. Ghosh (ed.) J. Roy (ed.) , Statistics: Applications and New Directions , Statistical Publishing Soc. (1984) pp. 504–511
[a11]	B.L.S. Prakasa Rao, "Asymptotic theory of statistical inference" , Wiley (1987)
[a12]	R. von Mises, "Wahrscheinlichkeitsrechnung" , Springer (1931)