# Best linear unbiased estimation in linear models

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
This article Best linear unbiased estimation in linear models was adapted from an original article by Simo Puntanen, George P.H. Styan, which appeared in StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies. The original article ([http://statprob.com/encyclopedia/BestLinearUnbiasedEstimatinInLinearModels.html StatProb Source], Local Files: pdf | tex) is copyrighted by the author(s), the article has been donated to Encyclopedia of Mathematics, and its further issues are under Creative Commons Attribution Share-Alike License'. All pages from StatProb are contained in the Category StatProb.

$\def\mx#1{ {\mathbf{#1}}}$ $\def\BETA{\beta}\def\BETAH{ {\hat\beta}}\def\BETAT{ {\tilde\beta}}\def\betat{\tilde\beta}$ $\def\C{ {\mathscr C}}$ $\def\cov{\mathrm{cov}}\def\M{ {\mathscr M}}$ $\def\NS{ {\mathscr N}}\def\OLSE{ {\small\mathrm{OLSE}}}$ $\def\rank{ {\rm rank}} \def\tr{ { \rm trace}}$ $\def\rz{ {\mathbf{R}}} \def\SIGMA{\Sigma} \def\var{ {\rm var}}$ $\def\BLUE}{\small\mathrm{BLUE}}$ $\def\BLUP}{\small\mathrm{BLUP}}$ $\def\EPS{\varepsilon}$ $\def\EE{E}$ $\def\E{E}$ $\def\GAMMA{\gamma}$ 2010 Mathematics Subject Classification: Primary: 62J05 [MSN][ZBL]

Best Linear Unbiased Estimation in Linear Models
Simo Puntanen 

University of Tampere, Finland

and

George P. H. Styan 

Keywords and Phrases: Best linear unbiased, BLUE, BLUP, Gauss--Markov Theorem, Generalized inverse, Ordinary least squares, OLSE.

## Introduction

In this article we consider the general linear model (Gauss--Markov model) \begin{equation*} \mx y = \mx X \BETA + \EPS, \quad \text{or in short } \M = \{ \mx y, \, \mx X \BETA, \, \sigma^2 \mx V \}, \end{equation*} where $\mx X$ is a known $n\times p$ model matrix, the vector $\mx y$ is an observable $n$-dimensional random vector, $\BETA$ is a $p\times 1$ vector of unknown parameters, and $\EPS$ is an unobservable vector of random errors with expectation $\EE(\EPS ) = \mx 0,$ and covariance matrix $\cov( \EPS) = \sigma^2 \mx V,$ where $\sigma^2 >0$ is an unknown constant. The nonnegative definite (possibly singular) matrix $\mx V$ is known. In our considerations $\sigma ^2$ has no role and hence we may put $\sigma^2=1.$

As regards the notation, we will use the symbols $\mx A',$ $\mx A^{-},$ $\mx A^{+},$ $\C(\mx A),$ $\C(\mx A)^{\bot},$ and $\NS(\mx A)$ to denote, respectively, the transpose, a generalized inverse, the Moore--Penrose inverse, the column space, the orthogonal complement of the column space, and the null space, of the matrix $\mx A$. By $(\mx A:\mx B)$ we denote the partitioned matrix with $\mx A$ and $\mx B$ as submatrices. By $\mx A^{\bot}$ we denote any matrix satisfying $\C(\mx A^{\bot}) = \NS(\mx A') = \C(\mx A)^{\bot}.$ Furthermore, we will write $\mx P_{\mx A} = \mx A\mx A^{+} = \mx A(\mx A'\mx A)^{-}\mx A'$ to denote the orthogonal projector (with respect to the standard inner product) onto $\C(\mx A).$ In particular, we denote $\mx{H} = \mx P_{\mx X}$ and $\mx{M} = \mx I_n - \mx H$. One choice for $\mx X^{\bot}$ is of course the projector $\mx M$.

Let $\mx K' \BETA$ be a given vector of parametric functions specified by $\mx K' \in \rz^{q\times p}.$ Our object is to find a (homogeneous) linear estimator $\mx A \mx y$ which would provide an unbiased and in some sense "best" estimator for $\mx K' \BETA$ under the model $\M.$ However, not all parametric functions have linear unbiased estimators; those which have are called estimable parametric functions, and then there exists a matrix $\mx A$ such that \begin{equation*} \E(\mx{Ay}) = \mx{AX}\BETA = \mx K' \BETA \quad \text{for all } \BETA \in \rz^p. \end{equation*} Hence $\mx{K}' \BETA$ is estimable if and only if there exists a matrix $\mx A$ such that $\mx{K}' = \mx{A}\mx{X}$, i.e., $\C(\mx K ) \subset \C(\mx X')$.

The ordinary least squares estimator of $\mx K' \BETA$ is defined as $\OLSE(\mx K' \BETA) = \mx K' \BETAH,$ where $\BETAH$ is any solution to the normal equation $\mx X' \mx X \BETAH = \mx X' \mx y$; hence $\BETA = \BETAH$ minimizes $(\mx y - \mx X\BETA)' (\mx y - \mx X\BETA)$ and it can be expressed as $\BETAH = (\mx X' \mx X) ^{-}\mx X' \mx y,$ while $\mx X\BETAH = \mx H \mx y.$ Now the condition $\C(\mx K ) \subset \C(\mx X')$ guarantees that $\mx K'\BETAH$ is unique, even though $\BETAH$ may not be unique.

## The Best Linear Unbiased Estimator (BLUE)

The expectation $\mx X\BETA$ is trivially estimable and $\mx{Gy}$ is unbiased for $\mx X\BETA$ whenever $\mx{G}\mx X = \mx{X}.$ An unbiased linear estimator $\mx{Gy}$ for $\mx X\BETA$ is defined to be the best linear unbiased estimator, $\BLUE$, for $\mx X\BETA$ under $\M$ if \begin{equation*} \cov( \mx{G} \mx y) \leq_{ {\rm L}} \cov( \mx{L} \mx y) \quad \text{for all } \mx{L} \colon \mx{L}\mx X = \mx{X}, \end{equation*} where "$\leq_\text{L}$" refers to the Löwner partial ordering. In other words, $\mx{G} \mx y$ has the smallest covariance matrix (in the Löwner sense) among all linear unbiased estimators. We denote the $\BLUE$ of $\mx X\BETA$ as $\BLUE(\mx X\BETA) = \mx X \BETAT.$ If $\mx X$ has full column rank, then $\BETA$ is estimable and an unbiased estimator $\mx A\mx y$ is the $\BLUE$ for $\BETA$ if $\mx{AVA}' \leq_{ {\rm L}} \mx{BVB}'$ for all $\mx{B}$ such that $\mx{BX} = \mx{I}_p.$ The Löwner ordering is a very strong ordering implying for example \begin{gather*} \var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p , \tr [\cov(\BETAT)] \le \tr [\cov(\BETA^{*})] , \qquad \det[\cov(\BETAT)] \le \det[\cov(\BETA^{*})], \end{gather*} for any linear unbiased estimator $\BETA^{*}$ of $\BETA$; here $\var$ refers to the variance and "det" denotes the determinant.

The following theorem gives the "Fundamental $\BLUE$ equation"; see, e.g., Rao (1967), Zyskind (1967) and Puntanen, Styan and Werner (2000).

Theorem 1. Consider the general linear model $\M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if $\mx G$ satisfies the equation $$\label{eq: 30jan09-fundablue} \mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}). \tag{1}$$ The corresponding condition for $\mx{Ay}$ to be the $\BLUE$ of an estimable parametric function $\mx{K}' \BETA$ is $\mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0})$.

It is sometimes convenient to express (1) in the following form, see Rao (1971).

Theorem 2.[Pandora's Box] Consider the general linear model $\M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if there exists a matrix $\mx{L} \in \rz^{p \times n}$ so that $\mx G$ is a solution to \begin{equation*} \begin{pmatrix} \mx V & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx G' \\ \mx{L} \end{pmatrix} = \begin{pmatrix} \mx 0 \\ \mx X' \end{pmatrix}. \end{equation*}

The equation (1) has a unique solution for $\mx G$ if and only if $\C(\mx X : \mx V) = \rz^n.$ Notice that under $\M$ we assume that the observed value of $\mx y$ belongs to the subspace $\C(\mx X : \mx V)$ with probability $1$; this is the consistency condition of the linear model, see, e.g., Baksalary, Rao and Markiewicz (1992). The consistency condition means, for example, that whenever we have some statements which involve the random vector $\mx y$, these statements need hold only for those values of $\mx y$ that belong to $\C(\mx{X}:\mx{V}).$ The general solution for $\mx G$ can be expressed, for example, in the following ways: $\mx G_1 = \mx{X}(\mx{X}'\mx{W}^{-}\mx{X})^{-}\mx{X}'\mx{W}^{-} + \mx F_{1}(\mx{I }_n - \mx W\mx W^{-} ) ,$ $\mx G_2 = \mx{H} - \mx{HVM}(\mx{MVM})^{-}\mx{M} + \mx F_{2}[\mx{I}_n - \mx{MVM}( \mx{MVM} )^{-} ]\mx M ,$ where $\mx F_{1}$ and $\mx F_{2}$ are arbitrary matrices, $\mx {W}= \mx V + \mx X\mx U\mx X'$ and $\mx U$ is any arbitrary conformable matrix such that $\C(\mx W) = \C(\mx X : \mx V).$ Notice that even though $\mx G$ may not be unique, the numerical value of $\mx G\mx y$ is unique because $\mx y \in \C(\mx X : \mx V).$ If $\mx V$ is positive definite, then $\BLUE(\mx X\BETA) = \mx X(\mx X' \mx V^{-1} \mx X)^{-} \mx X' \mx V^{-1} \mx y.$ Clearly $\OLSE(\mx X\BETA) = \mx H\mx y$ is the $\BLUE$ under $\{ \mx y, \, \mx X\BETA , \, \sigma^2\mx I \}.$ It is also worth noting that the matrix $\mx G$ satisfying (1) can be interpreted as a projector: it is a projector onto $\C(\mx X)$ along $\C(\mx V\mx X^{\bot}),$ see Rao (1974).

## OLSE vs. BLUE

Characterizing the equality of the Ordinary Least Squares Estimator $(\OLSE)$ and the $\BLUE$ has received a lot of attention in the literature, since Anderson (1948), but the major breakthroughs were made by Rao (1967) and Zyskind (1967); for a detailed review, see Puntanen and Styan (1989). For some further references from those years we may mention Kruskal (1968), Watson (1967), and Zyskind and Martin (1969).

We present below six characterizations for the $\OLSE$ and the $\BLUE$ to be equal (with probability $1$).

Theorem 3.[$\OLSE$ vs. $\BLUE$] Consider the general linear model $\M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ if and only if any one of the following six equivalent conditions holds. (Note: $\mx{V}$ may be replaced by its Moore--Penrose inverse $\mx{V}^+$ and $\mx{H}$ and $\mx{M} = \mx I_n - \mx H$ may be interchanged.)

(1) $\mx{HV} = \mx{VH}$,
(2) $\mx{H}\mx{V}\mx{M} = \mx{0}$,
(3) $\C(\mx{V}\mx{H})\subset\C(\mx H)$,
(4) $\C(\mx{X})$ has a basis comprising $r= \rank(\mx X)$ orthonormal eigenvectors of $\mx V$,
(5) $\mx{V} = \mx{HAH} + \mx{MBM}$ for some $\mx A$ and $\mx B$,
(6) $\mx{V} = \alpha\mx{I}_n + \mx{HKH} + \mx{M}\mx L\mx M$ for some $\alpha \in \rz$, and $\mx K$ and $\mx L$.

Theorem 3 shows at once that under $\{ \mx y, \, \mx X\BETA, \, \mx I_n \}$ the $\OLSE$ of $\mx X\BETA$ is trivially the $\BLUE$; this result is often called the Gauss--Markov Theorem.

## Two Linear Models

Consider now two linear models $\M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $\M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \}$, which differ only in their covariance matrices. For the proof of the following proposition and related discussion, see, e.g., Rao (1971, Th. 5.2, Th. 5.5), and Mitra and Moore (1973, Th. 3.3, Th. 4.1--4.2).

Theorem 4. Consider the linear models $\M_1 = \{\mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $\M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$ and let the notation $\{\BLUE(\mx X\BETA \mid \M_1) \} \subset \{\BLUE(\mx X\BETA \mid \M_2) \}$ mean that every representation of the $\BLUE$ for $\mx X\BETA$ under $\M_1$ remains the $\BLUE$ for $\mx X\BETA$ under $\M_2$. Then the following statements are equivalent:

(1) $\{ \BLUE(\mx X\BETA \mid \M_1) \} \subset \{ \BLUE(\mx X\BETA \mid \M_2) \}$,
(2) $\C(\mx V_2\mx X^{\bot}) \subset \C(\mx V_1 \mx X^\bot)$,
(3) $\mx V_2 = \mx V_1+ \mx{X} \mx N_1 \mx X' + \mx V_1\mx M\mx N_2 \mx M \mx V_1$, for some $\mx N_1$ and $\mx N_2$,
(4) $\mx V_2 = \mx{X} \mx N_3 \mx X' + \mx V_1\mx M\mx N_4 \mx M \mx V_1$, for some $\mx N_3$ and $\mx N_4$.

Notice that obviously \begin{align*} \{ \BLUE(\mx X \BETA \mid \M_1) \} = \{ \BLUE(\mx X \BETA \mid \M_2) \} \iff \C(\mx V_2\mx X^{\bot}) = \C(\mx V_1 \mx X^\bot). \end{align*} For the equality between the $\BLUE$s of $\mx X_1\BETA_1$ under two partitioned models, see Haslett and Puntanen (2010a).

## Model with New Observations: Best Linear Unbiased Predictor (BLUP)

Consider the model $\M = \{\mx y,\,\mx X\BETA,\,\mx V\},$ and let $\mx y_f$ denote an $m\times 1$ unobservable random vector containing new observations. The new observations are assumed to follow the linear model $\mx y_f = \mx X_f\BETA +\EPS_f ,$ where $\mx X_f$ is a known $m\times p$ model matrix associated with new observations, $\BETA$ is the same vector of unknown parameters as in $\M$, and $\EPS_f$ is an $m \times 1$ random error vector associated with new observations. Our goal is to predict the random vector $\mx y_f$ on the basis of $\mx y$. The expectation and the covariance matrix are \begin{equation*} \E\begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix} = \begin{pmatrix} \mx X\BETA \\ \mx X_f\BETA \end{pmatrix} , \quad \cov\begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix} = \begin{pmatrix} \mx V = \mx V_{11} & \mx{V}_{12} \\ \mx{V}_{21} & \mx V_{22} \end{pmatrix} , \end{equation*} which we may write as \begin{equation*} \M_f = \left \{ \begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix},\, \begin{pmatrix} \mx X\BETA \\ \mx X _f\BETA \end{pmatrix},\, \begin{pmatrix} \mx V & \mx{V}_{12} \\ \mx{V}_{21} & \mx V_{22} \end{pmatrix} \right \}. \end{equation*}

A linear predictor $\mx{Ay}$ is said to be unbiased for $\mx y_f$ if $\E(\mx{Ay}) = \E(\mx{y}_f) = \mx X_f\BETA$ for all $\BETA\in\rz^{p}.$ Then the random vector $\mx y_f$ is said to be unbiasedly predictable. Now an unbiased linear predictor $\mx{Ay}$ is the best linear unbiased predictor, $\BLUP$, for $\mx y_f$ if the Löwner ordering \begin{equation*} \cov(\mx{Ay}-\mx y_f) \leq_{ {\rm L}} \cov(\mx{By}-\mx y_f) \end{equation*} holds for all $\mx B$ such that $\mx{By}$ is an unbiased linear predictor for $\mx{y}_f$.

The following theorem characterizes the $\BLUP$; see, e.g., Christensen (2002, p. 283), and Isotalo and Puntanen (2006, p. 1015).

Theorem 5 (Fundamental $\BLUP$ equation) Consider the linear model $\M_f$, where $\mx{X}_f\BETA$ is a given estimable parametric function. Then the linear estimator $\mx{Ay}$ is the best linear unbiased predictor ($\BLUP$) for $\mx y_f$ if and only if $\mx{A}$ satisfies the equation \begin{equation*} \mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx{Ay}$ is the $\BLUP$ for $\mx y_f$ if and only if there exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \mx V & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \\ \mx L \end{pmatrix} = \begin{pmatrix} \mx{V}_{12} \\ \mx X_{f}' \end{pmatrix}. \end{equation*}

## The Mixed Model

A mixed linear model can be presented as \begin{equation*} \mx y = \mx X\BETA + \mx Z \GAMMA +\EPS , \quad \text{or shortly } \quad \M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \} , \end{equation*} where $\mx X \in \rz^{n \times p}$ and $\mx Z \in \rz^{n \times q}$ are known matrices, $\BETA \in \rz^{p}$ is a vector of unknown fixed effects, $\GAMMA$ is an unobservable vector ($q$ elements) of random effects with $\cov(\GAMMA,\EPS) = \mx 0_{q \times p}$ and \begin{equation*} \E(\GAMMA) = \mx 0_q , \quad \cov(\GAMMA) = \mx D_{q \times q}, \quad \E(\EPS) = \mx 0_n \,, \quad \cov(\EPS) = \mx R_{n\times n}. \end{equation*} This leads directly to:

Theorem 6. Consider the mixed model $\M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \}.$ Then the linear estimator $\mx B \mx y$ is the $\BLUE$ for $\mx X\BETA$ if and only if \begin{equation*} \mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) , \end{equation*} where $\SIGMA= \mx Z\mx D\mx Z' + \mx R$. Moreover, $\mx A \mx y$ is the $\BLUP$ for $\GAMMA$ if and only if \begin{equation*} \mx A(\mx X : \SIGMA \mx X^{\bot}) = (\mx 0 : \mx{D}\mx{Z}' \mx X^{\bot}). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx A \mx y = \BLUP(\GAMMA)$ if and only if there exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \SIGMA & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \\ \mx L \end{pmatrix} = \begin{pmatrix} \mx Z \mx D \\ \mx 0 \end{pmatrix}. \end{equation*}

For the equality between the $\BLUP$s under two mixed models, see Haslett and Puntanen (2010b, 2010c).

### Note

Reprinted with permission from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science+Business Media, LLC.

How to Cite This Entry:
Best linear unbiased estimation in linear models. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Best_linear_unbiased_estimation_in_linear_models&oldid=38515