Unbiased estimator

A statistical estimator whose expectation is that of the quantity to be estimated. Suppose that in the realization of a random variable taking values in a probability space , , a function has to be estimated, mapping the parameter set into a certain set , and that as an estimator of a statistic is chosen. If is such that

holds for , then is called an unbiased estimator of . An unbiased estimator is frequently called free of systematic errors.

Example 1.

Let be random variables having the same expectation , that is,

In that case the statistic

is an unbiased estimator of . In particular, the arithmetic mean of the observations, , is an unbiased estimator of . In this example .

Example 2.

Let be independent random variables having the same probability law with distribution function , that is,

In this case the empirical distribution function constructed from the observations is an unbiased estimator of , that is, , .

Example 3.

Let be an unbiased estimator of a parameter , that is, , and assume that is a linear function. In that case the statistic is an unbiased estimator of .

The next example shows that there are cases in which unbiased estimators exist and are even unique, but they may turn out to be useless.

Example 4.

Let be a random variable subject to the geometric distribution with parameter of success , that is, for any natural number ,

If is an unbiased estimator of the parameter , it must satisfy the unbiasedness equation , that is,

The unique solution of this equation is

Evidently, is good only when is very close to 1 or 0, otherwise carries no useful information on .

Example 5.

Suppose that a random variable has the binomial law with parameters and , that is, for any ,

It is known that the best unbiased estimator of the parameter (in the sense of minimum quadratic risk) is the statistic . Nevertheless, if is irrational, . This example reflects a general property of random variables that, generally speaking, a random variable need not take values that agree with its expectation. And finally, cases are possible when unbiased estimators do not exist at all. Thus, if under the conditions of Example 5 one takes as the function to be estimated , then (see Example 6) there is no unbiased estimator for .

The preceding examples demonstrate that the concept of an unbiased estimator in its very nature does not necessarily help an experimenter to avoid all the complications that arise in the construction of statistical estimators, since an unbiased estimator may turn out to be very good and even totally useless; it may not be unique or may not exist at all. Moreover, an unbiased estimator, like every point estimator, also has the following deficiency. It only gives an approximate value for the true value of the quantity to be estimated; this quantity was not known before the experiment and remains unknown after it has been performed. So, in the problem of constructing statistical point estimators there is no serious justification for the fact that in all cases they should produce the resulting unbiased estimator, unless it is assumed that the study of unbiased estimators leads to a simple priority theory. For example, the Rao–Cramér inequality has a simple form for unbiased estimators. Namely, if is an unbiased estimator for a function , then under fairly broad conditions of regularity on the family and the function , the Rao–Cramér inequality implies that

(1)

where is the Fisher amount of information for . Thus, there is a lower bound for the variance of an unbiased estimator of , namely, . In particular, if , then it follows from (1) that

A statistical estimator for which equality is attained in the Rao–Cramér inequality is called efficient (cf. Efficient estimator). Thus, the statistic in Example 5 is an efficient unbiased estimator of the parameter of the binomial law, since

and

that is, is the best point estimator of in the sense of minimum quadratic risk in the class of all unbiased estimators.

Naturally, an experimenter is interested in the case when the class of unbiased estimators is rich enough to allow the choice of the best unbiased estimator in some sense. In this context an important role is played by the Rao–Blackwell–Kolmogorov theorem, which allows one to construct an unbiased estimator of minimal variance. This theorem asserts that if the family has a sufficient statistic and is an arbitrary unbiased estimator of a function , then the statistic obtained by averaging over the fixed sufficient statistic has a risk not exceeding that of relative to any convex loss function for all . If the family is complete, the statistic is uniquely determined. That is, the Rao–Blackwell–Kolmogorov theorem implies that unbiased estimators must be looked for in terms of sufficient statistics, if they exist. The practical value of the Rao–Blackwell–Kolmogorov theorem lies in the fact that it gives a recipe for constructing best unbiased estimators, namely: One has to construct an arbitrary unbiased estimator and then average it over a sufficient statistic.

Example 6.

Suppose that a random variable has the Pascal distribution (a negative binomial distribution) with parameters and (, ); that is,

In this case the statistic is an unbiased estimator of . Since is expressed in terms of the sufficient statistic and the system of functions is complete on , is the only unbiased estimator and, consequently, the best estimator of .

Example 7.

Let be a random variable having the binomial law with parameters and . The generating function of this law can be expressed by the formula

which implies that for any integer , the -th derivative

On the other hand,

Hence,

that is, the statistic

(2)

is an unbiased estimator of , and since is expressed in terms of the sufficient statistic and the system of functions is complete on , it follows that is the only, hence the best, unbiased estimator of .

In connection with this example the following question arises: What functions of the parameter admit an unbiased estimator? A.N. Kolmogorov [1] has shown that this only happens for polynomials of degree . Thus, if

then it follows from (2) that the statistic

is the only unbiased estimator of . This result implies, in particular, that there is no unbiased estimator of .

Example 8.

Let be a random variable subject to the Poisson law with parameter ; that is, for any integer

Since , the observation of by itself is an unbiased estimator of its mathematical expectation . In turn, an unbiased estimator of, say, is . More generally, the statistic

is an unbiased estimator of . This fact implies, in particular, that the statistic

is an unbiased estimator of the function , . Quite generally, if admits an unbiased estimator, then the unbiasedness equation must hold for it, which is equivalent to

From this one deduces that an unbiased estimator exists for any function that admits a power series expansion in its domain of definition .

Example 9.

Suppose that the independent random variables have the same Poisson law with parameter , . The generating function of this law, which can be expressed by the formula

is an entire analytic function and hence has a unique unbiased estimator. In this case a sufficient statistic is , which has the Poisson law with parameter . If is an unbiased estimator of , then it must satisfy the unbiasedness equation

which implies that

that is, an unbiased estimator of the generating function of the Poisson law is the generating function of the binomial law with parameters and .

Examples 6–9 demonstrate that in certain cases, which occur quite frequently in practice, the problem of constructing best estimators is easily solvable, provided that one restricts attention to the class of unbiased estimators. Kolmogorov [1] has considered the problem of constructing unbiased estimators, in particular, for the distribution function of a normal law with unknown parameters. A more general definition of an unbiased estimator is due to E. Lehmann [2], according to whom a statistical estimator of a parameter is called unbiased relative to a loss function if

There is also a modification of this definition (see [3]). Yu.V. Linnik and his students (see [4]) have established that under fairly wide assumptions the best unbiased estimator is independent of the loss function.

References

[1]	A.N. Kolmogorov, "Unbiased estimates" Izv. Akad. Nauk SSSR Ser. Mat. , 14 : 4 (1950) pp. 303–326 (In Russian)
[2]	E.L. Lehmann, "Testing statistical hypotheses" , Wiley (1959)
[3]	L.B. Klebanov, "A general definition of unbiasedness" Theor. Probab. Appl. , 21 : 3 (1976) pp. 571–585 Teor. Veroyatnost. i. Primenen. , 21 : 3 (1976) pp. 584–598
[4]	L.B. Klebanov, Yu.V. Linnik, A.L. Rukhin, "Unbiased estimation and matrix loss functions" Soviet Math. Dokl. , 12 : 5 (1971) pp. 1526–1528 Dokl. Akad. Nauk SSSR , 200 : 5 (1971) pp. 1024–1025
[5]	S. Zacks, "The theory of statistical inference" , Wiley (1971)