Best linear unbiased estimation in linear models

Copyright notice

This article Best linear unbiased estimation in linear models was adapted from an original article by Simo Puntanen, George P.H. Styan, which appeared in StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies. The original article ([http://statprob.com/encyclopedia/BestLinearUnbiasedEstimatinInLinearModels.html StatProb Source], Local Files: pdf | tex) is copyrighted by the author(s), the article has been donated to Encyclopedia of Mathematics, and its further issues are under Creative Commons Attribution Share-Alike License'. All pages from StatProb are contained in the Category StatProb.

$\def\mx#1{ {\mathbf{#1}}}$ $\def\BETA{\beta}\def\BETAH{ {\hat\beta}}\def\BETAT{ {\tilde\beta}}\def\betat{\tilde\beta}$ $\def\C{ {\mathscr C}}$ $\def\cov{\mathrm{cov}}\def\M{ {\mathscr M}}$ $\def\NS{ {\mathscr N}}\def\OLSE{ {\small\mathrm{OLSE}}}$ $\def\rank{ {\rm rank}} \def\tr{ { \rm trace}}$ $\def\rz{ {\mathbf{R}}} \def\SIGMA{\Sigma} \def\var{ {\rm var}}$ $\def\BLUE}{\small\mathrm{BLUE}}$ $\def\BLUP}{\small\mathrm{BLUP}}$ $\def\EPS{\varepsilon}$ $\def\EE{E}$ $\def\E{E}$ $\def\GAMMA{\gamma}$ 2020 Mathematics Subject Classification: Primary: 62J05 [MSN][ZBL]

Best Linear Unbiased Estimation in Linear Models Simo Puntanen

University of Tampere, Finland

and

George P. H. Styan

McGill University, Montréal, Canada

Keywords and Phrases: Best linear unbiased, BLUE, BLUP, Gauss--Markov Theorem, Generalized inverse, Ordinary least squares, OLSE.

Introduction

In this article we consider the general linear model (Gauss--Markov model) \begin{equation*} \mx y = \mx X \BETA + \EPS, \quad \text{or in short } \M = \{ \mx y, \, \mx X \BETA, \, \sigma^2 \mx V \}, \end{equation*} where $\mx X$ is a known $n\times p$ model matrix, the vector $\mx y$ is an observable $n$-dimensional random vector, $\BETA$ is a $p\times 1$ vector of unknown parameters, and $\EPS$ is an unobservable vector of random errors with expectation $\EE(\EPS ) = \mx 0,$ and covariance matrix $\cov( \EPS) = \sigma^2 \mx V,$ where $\sigma^2 >0$ is an unknown constant. The nonnegative definite (possibly singular) matrix $\mx V $ is known. In our considerations $\sigma ^2$ has no role and hence we may put $\sigma^2=1.$

As regards the notation, we will use the symbols $\mx A',$ $\mx A^{-},$ $\mx A^{+},$ $\C(\mx A),$ $\C(\mx A)^{\bot},$ and $\NS(\mx A)$ to denote, respectively, the transpose, a generalized inverse, the Moore--Penrose inverse, the column space, the orthogonal complement of the column space, and the null space, of the matrix $\mx A$. By $(\mx A:\mx B)$ we denote the partitioned matrix with $\mx A$ and $\mx B$ as submatrices. By $\mx A^{\bot}$ we denote any matrix satisfying $\C(\mx A^{\bot}) = \NS(\mx A') = \C(\mx A)^{\bot}.$ Furthermore, we will write $\mx P_{\mx A} = \mx A\mx A^{+} = \mx A(\mx A'\mx A)^{-}\mx A'$ to denote the orthogonal projector (with respect to the standard inner product) onto $\C(\mx A).$ In particular, we denote $\mx{H} = \mx P_{\mx X}$ and $ \mx{M} = \mx I_n - \mx H$. One choice for $\mx X^{\bot}$ is of course the projector $\mx M$.

Let $\mx K' \BETA$ be a given vector of parametric functions specified by $\mx K' \in \rz^{q\times p}.$ Our object is to find a (homogeneous) linear estimator $\mx A \mx y$ which would provide an unbiased and in some sense "best" estimator for $\mx K' \BETA$ under the model $\M.$ However, not all parametric functions have linear unbiased estimators; those which have are called estimable parametric functions, and then there exists a matrix $\mx A$ such that \begin{equation*} \E(\mx{Ay}) = \mx{AX}\BETA = \mx K' \BETA \quad \text{for all } \BETA \in \rz^p. \end{equation*} Hence $\mx{K}' \BETA$ is estimable if and only if there exists a matrix $\mx A$ such that $\mx{K}' = \mx{A}\mx{X}$, i.e., $ \C(\mx K ) \subset \C(\mx X')$.

The ordinary least squares estimator of $\mx K' \BETA$ is defined as $ \OLSE(\mx K' \BETA) = \mx K' \BETAH, $ where $\BETAH$ is any solution to the normal equation $\mx X' \mx X \BETAH = \mx X' \mx y$; hence $\BETA = \BETAH$ minimizes $(\mx y - \mx X\BETA)' (\mx y - \mx X\BETA)$ and it can be expressed as $\BETAH = (\mx X' \mx X) ^{-}\mx X' \mx y,$ while $\mx X\BETAH = \mx H \mx y.$ Now the condition $\C(\mx K ) \subset \C(\mx X')$ guarantees that $\mx K'\BETAH$ is unique, even though $\BETAH$ may not be unique.

The Best Linear Unbiased Estimator (BLUE)

The expectation $\mx X\BETA$ is trivially estimable and $\mx{Gy}$ is unbiased for $\mx X\BETA$ whenever $ \mx{G}\mx X = \mx{X}.$ An unbiased linear estimator $\mx{Gy}$ for $\mx X\BETA$ is defined to be the best linear unbiased estimator, $\BLUE$, for $\mx X\BETA$ under $\M$ if \begin{equation*} \cov( \mx{G} \mx y) \leq_{ {\rm L}} \cov( \mx{L} \mx y) \quad \text{for all } \mx{L} \colon \mx{L}\mx X = \mx{X}, \end{equation*} where "$\leq_\text{L}$" refers to the Löwner partial ordering. In other words, $\mx{G} \mx y$ has the smallest covariance matrix (in the Löwner sense) among all linear unbiased estimators. We denote the $\BLUE$ of $\mx X\BETA$ as $ \BLUE(\mx X\BETA) = \mx X \BETAT. $ If $\mx X$ has full column rank, then $\BETA$ is estimable and an unbiased estimator $\mx A\mx y$ is the $\BLUE$ for $\BETA$ if $ \mx{AVA}' \leq_{ {\rm L}} \mx{BVB}' $ for all $\mx{B}$ such that $ \mx{BX} = \mx{I}_p. $ The Löwner ordering is a very strong ordering implying for example \begin{gather*} \var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p , \tr [\cov(\BETAT)] \le \tr [\cov(\BETA^{*})] , \qquad \det[\cov(\BETAT)] \le \det[\cov(\BETA^{*})], \end{gather*} for any linear unbiased estimator $\BETA^{*}$ of $\BETA$; here $\var$ refers to the variance and "det" denotes the determinant.

The following theorem gives the "Fundamental $\BLUE$ equation"; see, e.g., Rao (1967), Zyskind (1967) and Puntanen, Styan and Werner (2000).

Theorem 1. Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$ Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if $\mx G$ satisfies the equation $$ \label{eq: 30jan09-fundablue} \mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}). \tag{1}$$

The corresponding condition for $\mx{Ay}$ to be the $\BLUE$ of an estimable parametric function $\mx{K}' \BETA$ is $ \mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0}). $ .

It is sometimes convenient to express (1) in the following form, see Rao (1971).

Theorem 2.[Pandora's Box] Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$ Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if there exists a matrix $\mx{L} \in \rz^{p \times n}$ so that $\mx G$ is a solution to \begin{equation*} \begin{pmatrix} \mx V & \mx X \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx G' \mx{L} \end{pmatrix} = \begin{pmatrix} \mx 0 \mx X' \end{pmatrix}. \end{equation*} .

The equation (1) has a unique solution for $\mx G$ if and only if $\C(\mx X : \mx V) = \rz^n.$ Notice that under $\M$ we assume that the observed value of $\mx y$ belongs to the subspace $\C(\mx X : \mx V)$ with probability $1$; this is the consistency condition of the linear model, see, e.g., Baksalary, Rao and Markiewicz (1992). The consistency condition means, for example, that whenever we have some statements which involve the random vector~$\mx y$, these statements need hold only for those values of $\mx y$ that belong to $\C(\mx{X}:\mx{V}).$ The general solution for $\mx G$ can be expressed, for example, in the following ways: \[ \mx G_1 = \mx{X}(\mx{X}'\mx{W}^{-}\mx{X})^{-}\mx{X}'\mx{W}^{-} + \mx F_{1}(\mx{I }_n - \mx W\mx W^{-} ) , \] \[ \mx G_2 = \mx{H} - \mx{HVM}(\mx{MVM})^{-}\mx{M} + \mx F_{2}[\mx{I}_n - \mx{MVM}( \mx{MVM} )^{-} ]\mx M , \] where $\mx F_{1}$ and $\mx F_{2}$ are arbitrary matrices, $\mx {W}= \mx V + \mx X\mx U\mx X'$ and $\mx U$ is any arbitrary conformable matrix such that $\C(\mx W) = \C(\mx X : \mx V).$ Notice that even though $\mx G$ may not be unique, the numerical value of $\mx G\mx y$ is unique because $\mx y \in \C(\mx X : \mx V).$ If $\mx V$ is positive definite, then $\BLUE(\mx X\BETA) = \mx X(\mx X' \mx V^{-1} \mx X)^{-} \mx X' \mx V^{-1} \mx y.$ Clearly $\OLSE(\mx X\BETA) = \mx H\mx y$ is the $\BLUE$ under $\{ \mx y, \, \mx X\BETA , \, \sigma^2\mx I \}.$ It is also worth noting that the matrix $\mx G$ satisfying (1) can be interpreted as a projector: it is a projector onto $\C(\mx X)$ along $\C(\mx V\mx X^{\bot}),$ see Rao (1974).

OLSE vs. BLUE

Characterizing the equality of the Ordinary Least Squares Estimator $(\OLSE)$ and the $\BLUE$ has received a lot of attention in the literature, since Anderson (1948), but the major breakthroughs were made by Rao (1967) and Zyskind (1967); for a detailed review, see Puntanen and Styan (1989). For some further references from those years we may mention Kruskal (1968), Watson (1967), and Zyskind and Martin (1969).

We present below six characterizations for the $\OLSE$ and the $\BLUE$ to be equal (with probability $1$).

Theorem 3.[$\OLSE$ vs. $\BLUE$]\label{cor: olseblue}Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$ Then $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ if and only if any one of the following six equivalent conditions holds. $($Note: $\mx{V}$ may be replaced by its Moore--Penrose inverse $\mx{V}^+$ and $\mx{H}$ and $\mx{M} = \mx I_n - \mx H$ may be interchanged.$)$ \begin{align*} &{ {\rm (1)}} \; \mx{HV} = \mx{VH} , &{ {\rm (2)}} \; \mx{H}\mx{V}\mx{M} = \mx{0} \,, &{ {\rm (3)}} \; \C(\mx{V}\mx{H})\subset\C(\mx H) , &{ {\rm (4)}} \; \C(\mx{X}) \text{ has a basis comprising } r= \rank(\mx X) \text{ orthonormal eigenvectors of } \mx V, &{ {\rm (5)}}\; \mx{V} = \mx{HAH} + \mx{MBM} \text{ for some } \mx A \text{ and } \mx B, &{ {\rm (6)}}\; \mx{V} = \alpha\mx{I}_n + \mx{HKH} + \mx{M}\mx L\mx M \text{ for some } \alpha \in \rz \text{ , and } \mx K \text{ and } \mx L. \end{align*} .

Theorem 3 shows at once that under $\{ \mx y, \, \mx X\BETA, \, \mx I_n \}$ the $\OLSE$ of $\mx X\BETA$ is trivially the $\BLUE$; this result is often called the Gauss--Markov Theorem.

Two Linear Models

Consider now two linear models $ \M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $ \; \M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$ which differ only in their covariance matrices. For the proof of the following proposition and related discussion, see, e.g., Rao (1971, Th. 5.2, Th. 5.5), and Mitra and Moore (1973, Th. 3.3, Th. 4.1--4.2).

Theorem 4. \label{propo: 1111} Consider the linear models $ \M_1 = \{\mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $ \M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$ and let the notation $ \{\BLUE(\mx X\BETA \mid \M_1) \} \subset \{\BLUE(\mx X\BETA \mid \M_2) \} $ mean that every representation of the $\BLUE$ for $\mx X\BETA$ under $\M_1$ remains the $\BLUE$ for $\mx X\BETA$ under $\M_2$. Then the following statements are equivalent: \begin{align*} &{ {\rm (1)}} \; \{ \BLUE(\mx X\BETA \mid \M_1) \} \subset \{ \BLUE(\mx X\BETA \mid \M_2) \} , &{ {\rm (2)}} \; \C(\mx V_2\mx X^{\bot}) \subset \C(\mx V_1 \mx X^\bot) , &{ {\rm (3)}} \; \mx V_2 = \mx V_1+ \mx{X} \mx N_1 \mx X' + \mx V_1\mx M\mx N_2 \mx M \mx V_1, \text{ for some } \mx N_1 \text{ and } \mx N_2, &{ {\rm (4)}} \; \mx V_2 = \mx{X} \mx N_3 \mx X' + \mx V_1\mx M\mx N_4 \mx M \mx V_1, \text{ for some } \mx N_3 \text{ and } \mx N_4. \end{align*}

.

Notice that obviously \begin{align*} \{ \BLUE(\mx X \BETA \mid \M_1) \} = \{ \BLUE(\mx X \BETA \mid \M_2) \} \iff \C(\mx V_2\mx X^{\bot}) = \C(\mx V_1 \mx X^\bot). \end{align*} For the equality between the $\BLUE$s of $\mx X_1\BETA_1$ under two partitioned models, see Haslett and Puntanen (2010a).

Model with New Observations: Best Linear Unbiased Predictor (BLUP)

Consider the model $ \M = \{\mx y,\,\mx X\BETA,\,\mx V\},$ and let $\mx y_f$ denote an $m\times 1$ unobservable random vector containing new observations. The new observations are assumed to follow the linear model $ \mx y_f = \mx X_f\BETA +\EPS_f ,$ where $\mx X_f$ is a known $m\times p$ model matrix associated with new observations, $\BETA$ is the same vector of unknown parameters as in $\M$, and $\EPS_f$ is an $m \times 1$ random error vector associated with new observations. Our goal is to predict the random vector $\mx y_f$ on the basis of $\mx y$. The expectation and the covariance matrix are \begin{equation*} \E\begin{pmatrix} \mx y \mx y_f \end{pmatrix} = \begin{pmatrix} \mx X\BETA \mx X_f\BETA \end{pmatrix} , \quad \cov\begin{pmatrix} \mx y \mx y_f \end{pmatrix} = \begin{pmatrix} \mx V = \mx V_{11} & \mx{V}_{12} \mx{V}_{21} & \mx V_{22} \end{pmatrix} , \end{equation*} which we may write as \begin{equation*} \M_f = \left \{ \begin{pmatrix} \mx y \mx y_f \end{pmatrix},\, \begin{pmatrix} \mx X\BETA \mx X _f\BETA \end{pmatrix},\, \begin{pmatrix} \mx V & \mx{V}_{12} \mx{V}_{21} & \mx V_{22} \end{pmatrix} \right \}. \end{equation*}

A linear predictor $\mx{Ay}$ is said to be unbiased for $\mx y_f$ if $ \E(\mx{Ay}) = \E(\mx{y}_f) = \mx X_f\BETA$ for all $\BETA\in\rz^{p}.$ Then the random vector $\mx y_f$ is said to be unbiasedly predictable. Now an unbiased linear predictor $\mx{Ay}$ is the best linear unbiased predictor, $\BLUP$, for $\mx y_f$ if the Löwner ordering \begin{equation*} \cov(\mx{Ay}-\mx y_f) \leq_{ {\rm L}} \cov(\mx{By}-\mx y_f) \end{equation*} holds for all $\mx B$ such that $\mx{By}$ is an unbiased linear predictor for $\mx{y}_f$.

The following theorem characterizes the $\BLUP$; see, e.g., Christensen (2002, p. 283), and Isotalo and Puntanen (2006, p. 1015).

Theorem 5.[Fundamental $\BLUP$ equation]\label{propo:BLUP-funda} Consider the linear model $\M_f,$ where $\mx{X}_f\BETA$ is a given estimable parametric function. Then the linear estimator $\mx{Ay}$ is the best linear unbiased predictor { {\rm (}}$\BLUP${ {\rm )}} for $\mx y_f$ if and only if $\mx{A}$ satisfies the equation \begin{equation*} \mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx{Ay}$ is the $\BLUP$ for $\mx y_f$ if and only if there exists a matrix~$\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \mx V & \mx X \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \mx L \end{pmatrix} = \begin{pmatrix} \mx{V}_{12} \mx X_{f}' \end{pmatrix}. \end{equation*} .

The Mixed Model

A mixed linear model can be presented as \begin{equation*} \mx y = \mx X\BETA + \mx Z \GAMMA +\EPS , \quad \text{or shortly } \quad \M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \} , \end{equation*} where $\mx X \in \rz^{n \times p}$ and $\mx Z \in \rz^{n \times q}$ are known matrices, $\BETA \in \rz^{p}$ is a vector of unknown fixed effects, $\GAMMA$ is an unobservable vector ($q$ elements) of random effects with $\cov(\GAMMA,\EPS) = \mx 0_{q \times p}$ and \begin{equation*} \E(\GAMMA) = \mx 0_q , \quad \cov(\GAMMA) = \mx D_{q \times q}, \quad \E(\EPS) = \mx 0_n \,, \quad \cov(\EPS) = \mx R_{n\times n}. \end{equation*} This leads directly to:

Theorem 6. Consider the mixed model $ \M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \}.$ Then the linear estimator $\mx B \mx y$ is the $\BLUE$ for $\mx X\BETA$ if and only if \begin{equation*} \mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) , \end{equation*} where $\SIGMA= \mx Z\mx D\mx Z' + \mx R$. Moreover, $\mx A \mx y$ is the $\BLUP$ for $\GAMMA$ if and only if \begin{equation*} \mx A(\mx X : \SIGMA \mx X^{\bot}) = (\mx 0 : \mx{D}\mx{Z}' \mx X^{\bot}). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx A \mx y = \BLUP(\GAMMA)$ if and only if there exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \SIGMA & \mx X \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \mx L \end{pmatrix} = \begin{pmatrix} \mx Z \mx D \mx 0 \end{pmatrix}. \end{equation*} .

For the equality between the $\BLUP$s under two mixed models, see Haslett and Puntanen (2010b, 2010c).

Note

Reprinted with permission from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science+Business Media, LLC.

References

[1]	Anderson, T. W. (1948). On the theory of testing serial correlation. Skandinavisk Aktuarietidskrift, 31, 88--116. \
[2]	Baksalary, Jerzy K.; Rao, C. Radhakrishna and Markiewicz, Augustyn (1992). A study of the influence of the `natural restrictions' on estimation problems in the singular {G}auss--{M}arkov model, Journal of Statistical Planning and Inference, 31, 335--351. \
[3]	Christensen, Ronald (2002). Plane Answers to Complex Questions: The Theory of Linear Models, 3rd Edition. Springer, New York. \
[4]	Haslett, Stephen J. and Puntanen, Simo (2010a). Effect of adding regressors on the equality of the BLUEs under two linear models. Journal of Statistical Planning and Inference, 140, 104--110, \
[5]	Haslett, Stephen J. and Puntanen, Simo (2010b). Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Statistical Papers, 51, 465--475. \
[6]	Haslett, Stephen J. and Puntanen, Simo (2010c). On the equality of the BLUPs under two linear mixed models. Metrika, available online, DOI 10.1007/s00184-010-0308-6. \
[7]	Isotalo, Jarkko and Puntanen, Simo (2006). Linear prediction sufficiency for new observations in the general Gauss--Markov model. Communications in Statistics: Theory and Methods, 35, 1011--1023. \
[8]	Kruskal, William (1967). When are Gauss--Markov and least squares estimators identical? {A} coordinate-free approach. The Annals of Mathematical Statistics, 39, 70--75. \
[9]	Mitra, Sujit Kumar and Moore, Betty Jeanne (1973). Gauss--Markov estimation with an incorrect dispersion matrix. Sankhya, Series~A, 35, 139--152. \
[10]	Puntanen, Simo and Styan, George P. H. (1989). The equality of the ordinary least squares estimator and the best linear unbiased estimator [with comments by Oscar Kempthorne and by Shayle~R. Searle and with ``Reply by the authors]. The American Statistician, 43, 153--164. \
[11]	Puntanen, Simo; Styan, George P. H. and Werner, Hans Joachim (2000). Two matrix-based proofs that the linear estimator Gy is the best linear unbiased estimator. Journal of Statistical Planning and Inference, 88, 173--179. \
[12]	Rao, C. Radhakrishna (1967). Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, 1965/1966, vol. 1 (Eds. Lucien~M. Le Cam & Jerzy Neyman), University of California Press, Berkeley, pp. 355--372. \
[13]	Rao, C. Radhakrishna (1971). Unified theory of linear estimation. Sankhya, Series~A, 33, 371--394. [Corrigenda (1972), 34, p. 194 and p. 477.] \
[14]	Rao, C. Radhakrishna (1974). Projectors, generalized inverses and the BLUE's. Journal of the Royal Statistical Society, Series~B, 36, 442--448. \
[15]	Watson, Geoffrey S. (1967). Linear least squares regression. \emph {The Annals of Mathematical Statistics}, 38, 1679--1699. \
[16]	Zyskind, George (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. The Annals of Mathematical Statistics, 38, 1092--1109. \
[17]	Zyskind, George and Martin, Frank B. (1969). On best linear estimation and general Gauss--Markov theorem in linear models with arbitrary nonnegative covariance structure. SIAM Journal on Applied Mathematics, 17, 1190--1202.