Linear estimator

A linear function of observable random variables, used (when the actual values of the observed variables are substituted into it) as an approximate value (estimate) of an unknown parameter of the stochastic model under analysis (see Statistical estimator). The special selection of the class of linear estimators is justified for the following reasons. Linear estimators lend themselves more easily to statistical analysis, in particular to the investigation of consistency, unbiasedness, efficiency, the construction of corresponding confidence intervals, etc. At the same time, in a fairly wide range of cases the search for "optimal" (in a well-defined sense) estimators does not lead beyond the limits of the class of linear estimators. For example, the statistical analysis of a linear regression model (see Linear regression) of the form

$$ \mathbf Y = \mathbf X \pmb\theta + \epsilon $$

gives as best linear unbiased estimator of the parameter $ \pmb\theta $ the least-squares estimator

$$ \widehat{ {\pmb\theta }} = \ ( \mathbf X ^ \prime \mathbf X )^{-1} \mathbf X ^ \prime \mathbf Y $$

(linear with respect to the observed values of the random variable $ \mathbf Y $ under investigation). Here $ \mathbf Y $ is the $ n $- dimensional column vector of observed values $ y _ {i} $, $ i = 1 \dots n $, of the resulting test (random variable) under investigation, $ \mathbf X $ is the $ n \times p $ matrix (of rank $ p $) of observed values $ x _ {i}^{(k)} $, $ i = 1 \dots n $, $ k = 1 \dots p $, of $ p $ non-random factor arguments on which the resulting test $ \mathbf Y $ depends, $ \pmb\theta $ is the $ p $- dimensional column vector of the unknown parameters $ Q _ {k} $, $ k = 1 \dots p $, and $ \epsilon $ is the $ n $- dimensional random column vector of residual components, which satisfies the condition $ {\mathsf E} \epsilon = 0 $, $ {\mathsf E} ( \epsilon \epsilon^\prime ) = \sigma^{2} I $( $ I $ being the unit matrix).

References

[1]	H. Cramér, "Mathematical methods of statistics" , Princeton Univ. Press (1946)
[2]	C.R. Rao, "Linear statistical inference and its applications" , Wiley (1965)
[3]	S. Zacks, "The theory of statistical inference" , Wiley (1971)
[4]	H. Scheffé, "Analysis of variance" , Wiley (1959)

Comments

When $ \epsilon $ is multivariate normally distributed, the least-squares estimator is actually optimal in a wider class of estimators: it is the minimum-variance unbiased estimator. Even without the normality assumption it is asymptotically efficient ( $ n \rightarrow \infty $) (cf. Efficiency, asymptotic) among all estimators.