View source for Bayesian approach, empirical

<!--
b0154001.png
$#A+1 = 101 n = 0
$#C+1 = 101 : ~/encyclopedia/old_files/data/B015/B.0105400 Bayesian approach, empirical
Automatically converted into TeX, above some diagnostics.
Please remove this comment and the {{TEX|auto}} line below,
if TeX found to be correct.
-->

{{TEX|auto}}
{{TEX|done}}

A statistical interpretation of the [[Bayesian approach|Bayesian approach]] yielding conclusions on unobservable parameters even if their [[A priori distribution|a priori distribution]] is unknown. Let  $  (Y, X) $
be a random vector for which the density  $  p(y \mid  x) $
of the conditional distribution of  $  Y $
for any given value of the random parameter  $  X = x $
is known. If, as a result of some experiment, only the realization of  $  Y $
is observed, while the corresponding realization of  $  X $
is unknown, and if it is necessary to estimate the value of a given function  $  \phi (X) $
of the non-observed realization, then, in accordance with the empirical Bayesian approach, the conditional mathematical expectation  $  {\mathsf E} \{ \phi (X) \mid  Y \} $
should be used as an approximate value  $  \psi (Y) $
for  $  \phi (X) $. 
In view of the [[Bayes formula|Bayes formula]], this expectation is given by the formula

$$ \tag{1 }
\psi (Y)  = \ 

\frac{\int\limits \phi (x) p (Y \mid  x) p (x)  d \mu (x) }{q (Y) }
 ,
$$

where

$$ \tag{2 }
q (y)  =  \int\limits p (y \mid  x) p (x)  d \mu (x),
$$

$  p(x) $
is the density of the unconditional (a priori) distribution of  $  X $, 
$  \mu (x) $
is the corresponding  $  \sigma $-
finite measure; and the function  $  q(y) $
represents the density of the unconditional distribution of  $  Y $.

If the a priori density  $  p(x) $
is unknown, it is not possible to compute the values of  $  \psi $
and  $  q $. 
However, if a sufficiently large number of realizations of the random variables  $  Y _ {1} \dots Y _ {k} $, 
which are drawn from the distribution with density  $  q(y) $, 
is known, it is possible to construct a consistent estimator  $  \widehat{q}  (y) $, 
which depends only on  $  Y _ {1} \dots Y _ {k} $. 
S.N. Bernshtein [[#References|[1]]] proposed to estimate the value of  $  \psi (Y) $
by substituting  $  \widehat{q}  (y) $
for  $  q(y) $
in (2), finding the solution  $  \widehat{p}  (x) $
of this integral equation, and then substituting  $  \widehat{p}  $
and  $  \widehat{q}  $
in the right-hand side of (1). However, this method is difficult, since solving this integral equation (2) is an ill-posed problem in numerical mathematics.

In certain special cases the statistical approach may be employed not only to estimate  $  q $, 
but also  $  \psi $[[#References|[3]]]. This is possible if the identity

$$ \tag{3 }
\phi (x)p(y \mid  x)  = \ 
\lambda (y) r [z(y) \mid  x ] ,
$$

involving  $  x $
and  $  y $, 
is true. In (3),  $  \lambda (y) $
and  $  z(y) $
are functions which depend on  $  y $
only, while  $  r(z \mid  x) $, 
being a function of  $  z $, 
is a probability density (i.e. may be regarded as the density of an arbitrary distribution of some random variable  $  Z $
for a given value  $  X = x $). 
If (3) is true, the numerator of (1) is equal to the product  $  \lambda (Y)s[z(Y)] $, 
where  $  s(z) = \int r(z \mid  x)p(x)  d \mu (x) $
is the density of the unconditional distribution of  $  Z $. 
Thus, if a sufficiently large number of realizations of independent random variables  $  Z _ {1} \dots Z _ {m} $
with density distribution  $  s(z) $
is available, then it is possible to construct a consistent estimator  $  \widehat{s}  (z) $
for  $  s (z) $, 
and hence also to find a consistent estimator  $  \widehat \psi   (Y) $
for  $  \psi (Y) $:

$$ \tag{4 }
\phi (X)  \approx  \psi (Y)  \approx \ 
\widehat \psi   (Y)  = \ 

\frac{\lambda (Y) \widehat{s}  [z (Y)] }{\widehat{q}  (Y) }
 .
$$

For instance, if one has to estimate  $  \psi (X) = X  ^ {h} $, 
where  $  h $
is a positive integer, and  $  p(y \mid  x) = x  ^ {y} e  ^ {-x} / y ! $(
$  y = 0, 1 ,\dots $; 
$  x > 0 $), 
then  $  \phi (x) p (y \mid  x) = \lambda (y)p(y + h \mid  x) $, 
where  $  \lambda (y) = (y+h) ! /y ! $. 
Since, here,  $  r(z \mid  x) = p(z \mid  x) $, 
one has  $  s(z) = q(z) $. 
Accordingly,  $  \widehat \psi   (Y) = \lambda (Y) \widehat{q}  (Y+h)/ \widehat{q}  (Y) $, 
i.e. only the sequence of realizations  $  Y _ {1} , Y _ {2} \dots $
is required to find  $  \widehat \psi   $. 
If, on the other hand,  $  p(y \mid  x) = b (y \mid  x) = C _ {n}  ^ {y} x  ^ {y} (1 - x) ^ {n-y } $(
$  y = 0 \dots n $; 
$  n $
is a positive integer;  $  0 \leq  x \leq  1 $, 
$  C _ {n}  ^ {y} = ( {} _ {y}  ^ {n} ) $), 
then  $  \psi (x)p(y \mid  x) = \lambda (y)r(y+h \mid  x) $, 
where  $  \lambda (y) = C _ {n}  ^ {y} / C _ {n+h }  ^ {y+h } $
and  $  r(z \mid  x) = b _ {n+h }  (z \mid  x) \neq p(z \mid  x) $. 
For this reason two sequences of empirical values  $  Y _ {i} $
and  $  Z _ {j} $
are required in this case to construct  $  \widehat \psi   (Y) $.

This form of the empirical Bayesian approach is applicable to the very narrow class of densities  $  p(y \mid  x) $
and functions  $  \phi (x) $
which satisfy condition (3); even if this condition is in fact met, the construction of the estimator (4) is subject to the observability of the random variables  $  Z _ {j} $, 
the distribution of which usually differs from that of the variables  $  Y _ {i} $
which are observed directly. For practical purposes, it is preferable to use the empirical Bayesian approach in a modified form, in which these disadvantages are absent. In this modification the approximation which is constructed does not yield a consistent estimator of  $  \psi (Y) $(
such an estimator may even be non-existent), but rather upper and lower estimators of this function, which are found by solving a problem of linear programming, as follows. Let  $  \Psi _ {1} (Y) $
and  $  \Psi _ {2} (Y) $
be the constrained minimum and maximum of the linear functional (with respect to the unknown a priori density  $  p(x) $) 
in the numerator of (1), calculated under the linear constraints  $  p(x) \geq  0 $, 
$  \int p(x)  d \mu (x) = 1 $
and  $  q(Y) \equiv \int p(Y \mid  x)p(x)  d \mu (x) = \widehat{q}  (Y) $, 
where  $  \widehat{q}  (Y) $
is the estimator of  $  q(Y) $
mentioned above, constructed from the results of the observations  $  Y _ {1} \dots Y _ {k} $. 
One may conclude in such a case that  $  \Psi _ {1} (Y)/ \widehat{q}  (Y) \leq  \psi (Y) \leq  \Psi _ {2} (Y)/ \widehat{q}  (Y) $, 
where the probability of the truth of this conclusion tends to one (by virtue of the [[Law of large numbers|law of large numbers]]) as the number of random variables  $  Y _ {i} $, 
used to construct the estimator  $  \widehat{q}  (Y) $, 
increases without limit. Other modifications of the empirical Bayesian approach are also possible — for example, by adding to the last-named condition  $  q(Y) = \widehat{q}  (Y) $
a finite number of conditions of the form  $  q( y _ {i} ) = \widehat{q}  (y _ {i} ) $, 
where  $  y _ {i} $
are preliminarily given numbers; if  $  \widehat{q}  $
is replaced by the corresponding confidence bounds for  $  q $, 
the conditions are obtained in the form of inequalities  $  q _ {1} (y _ {i} ) \leq  q(y _ {i} ) \leq  q _ {2} (y _ {i} ) $, 
etc.

In certain cases, which are important in practice, satisfactory majorants, which can be computed without the use of the laborious method of linear programming, can be found for the functions  $  \Psi _ {1} $
and  $  \Psi _ {2} $(
see the example in the entry [[Sample method|Sample method]] which deals with statistical control).

See the entry [[Discriminant analysis|Discriminant analysis]] for the applications of the empirical Bayesian approach to hypotheses testing concerning the values of random parameters.

====References====
<table><TR><TD valign="top">[1]</TD> <TD valign="top">  S.N. Bernshtein,   "On  "fiducial"  probabilities of Fisher"  ''Izv. Akad. Nauk SSSR Ser. Mat.'' , '''5'''  (1941)  pp. 85–94  (In Russian)  (English abstract)</TD></TR><TR><TD valign="top">[2]</TD> <TD valign="top">  L.N. Bol'shev,   "Applications of the empirical Bayes approach" , ''Proc. Internat. Congress Mathematicians (Nice, 1970)'' , '''3''' , Gauthier-Villars  (1971)  pp. 241–247</TD></TR><TR><TD valign="top">[3]</TD> <TD valign="top">  H. Robbins,   "An empirical Bayes approach to statistics" , ''Proc. Berkeley Symp. Math. Statist. Probab.'' , '''1''' , Berkeley-Los Angeles  (1956)  pp. 157–163</TD></TR></table>