Bayesian approach, empirical

A statistical interpretation of the Bayesian approach yielding conclusions on unobservable parameters even if their a priori distribution is unknown. Let be a random vector for which the density of the conditional distribution of for any given value of the random parameter is known. If, as a result of some experiment, only the realization of is observed, while the corresponding realization of is unknown, and if it is necessary to estimate the value of a given function of the non-observed realization, then, in accordance with the empirical Bayesian approach, the conditional mathematical expectation should be used as an approximate value for . In view of the Bayes formula, this expectation is given by the formula

(1)

where

(2)

is the density of the unconditional (a priori) distribution of , is the corresponding -finite measure; and the function represents the density of the unconditional distribution of .

If the a priori density is unknown, it is not possible to compute the values of and . However, if a sufficiently large number of realizations of the random variables , which are drawn from the distribution with density , is known, it is possible to construct a consistent estimator , which depends only on . S.N. Bernshtein [1] proposed to estimate the value of by substituting for in (2), finding the solution of this integral equation, and then substituting and in the right-hand side of (1). However, this method is difficult, since solving this integral equation (2) is an ill-posed problem in numerical mathematics.

In certain special cases the statistical approach may be employed not only to estimate , but also [3]. This is possible if the identity

(3)

involving and , is true. In (3), and are functions which depend on only, while , being a function of , is a probability density (i.e. may be regarded as the density of an arbitrary distribution of some random variable for a given value ). If (3) is true, the numerator of (1) is equal to the product , where is the density of the unconditional distribution of . Thus, if a sufficiently large number of realizations of independent random variables with density distribution is available, then it is possible to construct a consistent estimator for , and hence also to find a consistent estimator for :

(4)

For instance, if one has to estimate , where is a positive integer, and (; ), then , where . Since, here, , one has . Accordingly, , i.e. only the sequence of realizations is required to find . If, on the other hand, (; is a positive integer; , ), then , where and . For this reason two sequences of empirical values and are required in this case to construct .

This form of the empirical Bayesian approach is applicable to the very narrow class of densities and functions which satisfy condition (3); even if this condition is in fact met, the construction of the estimator (4) is subject to the observability of the random variables , the distribution of which usually differs from that of the variables which are observed directly. For practical purposes, it is preferable to use the empirical Bayesian approach in a modified form, in which these disadvantages are absent. In this modification the approximation which is constructed does not yield a consistent estimator of (such an estimator may even be non-existent), but rather upper and lower estimators of this function, which are found by solving a problem of linear programming, as follows. Let and be the constrained minimum and maximum of the linear functional (with respect to the unknown a priori density ) in the numerator of (1), calculated under the linear constraints , and , where is the estimator of mentioned above, constructed from the results of the observations . One may conclude in such a case that , where the probability of the truth of this conclusion tends to one (by virtue of the law of large numbers) as the number of random variables , used to construct the estimator , increases without limit. Other modifications of the empirical Bayesian approach are also possible — for example, by adding to the last-named condition a finite number of conditions of the form , where are preliminarily given numbers; if is replaced by the corresponding confidence bounds for , the conditions are obtained in the form of inequalities , etc.

In certain cases, which are important in practice, satisfactory majorants, which can be computed without the use of the laborious method of linear programming, can be found for the functions and (see the example in the entry Sample method which deals with statistical control).

See the entry Discriminant analysis for the applications of the empirical Bayesian approach to hypotheses testing concerning the values of random parameters.

References

[1]	S.N. Bernshtein, "On "fiducial" probabilities of Fisher" Izv. Akad. Nauk SSSR Ser. Mat. , 5 (1941) pp. 85–94 (In Russian) (English abstract)
[2]	L.N. Bol'shev, "Applications of the empirical Bayes approach" , Proc. Internat. Congress Mathematicians (Nice, 1970) , 3 , Gauthier-Villars (1971) pp. 241–247
[3]	H. Robbins, "An empirical Bayes approach to statistics" , Proc. Berkeley Symp. Math. Statist. Probab. , 1 , Berkeley-Los Angeles (1956) pp. 157–163