Namespaces
Variants
Actions

Difference between revisions of "Best linear unbiased estimation in linear models"

From Encyclopedia of Mathematics
Jump to: navigation, search
(accents)
 
(17 intermediate revisions by 2 users not shown)
Line 52: Line 52:
 
<ref> Department of Mathematics and Statistics,
 
<ref> Department of Mathematics and Statistics,
 
McGill University, 805 ouest rue Sherbrooke
 
McGill University, 805 ouest rue Sherbrooke
Street West, Montr&eacute;al (Qu&eacute;bec), Canada H3A 2K6.
+
Street West, Montréal (Québec), Canada H3A 2K6.
 
Email:  styan@math.mcgill.ca </ref>
 
Email:  styan@math.mcgill.ca </ref>
  
  
McGill University, Montr&eacute;al, Canada
+
McGill University, Montréal, Canada
 
</center>
 
</center>
 
<!-- \maketitle  
 
<!-- \maketitle  
Line 162: Line 162:
 
\mx{L}\mx X = \mx{X},
 
\mx{L}\mx X = \mx{X},
 
\end{equation*}
 
\end{equation*}
where "$\leq_\text{L}$" refers to the L&ouml;wner partial ordering.
+
where "$\leq_\text{L}$" refers to the Löwner partial ordering.
 
In other words, $\mx{G} \mx y$ has the smallest covariance matrix
 
In other words, $\mx{G} \mx y$ has the smallest covariance matrix
(in the L&ouml;wner sense) among all linear unbiased estimators.
+
(in the Löwner sense) among all linear unbiased estimators.
 
We denote the $\BLUE$ of $\mx X\BETA$ as
 
We denote the $\BLUE$ of $\mx X\BETA$ as
 
$ \BLUE(\mx X\BETA) = \mx X \BETAT. $
 
$ \BLUE(\mx X\BETA) = \mx X \BETAT. $
Line 172: Line 172:
 
$ for all $\mx{B}$ such that
 
$ for all $\mx{B}$ such that
 
$ \mx{BX} = \mx{I}_p. $
 
$ \mx{BX} = \mx{I}_p. $
The L&ouml;wner ordering is a very strong ordering implying for example
+
The Löwner ordering is a very strong ordering implying for example
 
\begin{gather*}
 
\begin{gather*}
 
\var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p ,  
 
\var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p ,  
Line 189: Line 189:
 
and Puntanen, Styan and Werner (2000).
 
and Puntanen, Styan and Werner (2000).
  
'''Theorem 1'''. Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$
+
'''Theorem 1'''. ''Consider the general linear model'' $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$.
Then
+
''Then the estimator'' $\mx{Gy}$
the estimator $\mx{Gy}$
+
''is the'' $\BLUE$ ''for'' $\mx X\BETA$ ''if and only if'' $\mx G$
is the $\BLUE$ for $\mx X\BETA$ if and only if $\mx G$
+
''satisfies the equation''
satisfies the equation
 
 
$$ \label{eq: 30jan09-fundablue}
 
$$ \label{eq: 30jan09-fundablue}
 
\mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}).
 
\mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}).
 
\tag{1}$$
 
\tag{1}$$
 
+
''The corresponding condition for'' $\mx{Ay}$ ''to be the'' $\BLUE$ ''of an estimable parametric function'' $\mx{K}' \BETA$ ''is'' $ \mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0})$.
The corresponding
 
condition for $\mx{Ay}$ to be the $\BLUE$ of an estimable parametric
 
function $\mx{K}' \BETA$
 
is
 
$ \mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0}). $
 
.
 
  
 
It is sometimes convenient to express
 
It is sometimes convenient to express
Line 211: Line 204:
  
 
'''Theorem 2'''.[Pandora's Box]
 
'''Theorem 2'''.[Pandora's Box]
Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$
+
''Consider the general linear model'' $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. ''Then the estimator'' $\mx{Gy}$ ''is the'' $\BLUE$ ''for'' $\mx X\BETA$ ''if and only if there exists a matrix'' $\mx{L} \in \rz^{p \times n}$ ''so that'' $\mx G$ ''is a solution to''
Then
 
the estimator $\mx{Gy}$
 
is the $\BLUE$ for $\mx X\BETA$ if and only if there exists
 
a matrix $\mx{L} \in \rz^{p \times n}$
 
so that $\mx G$ is a solution to
 
 
\begin{equation*}
 
\begin{equation*}
 
\begin{pmatrix}
 
\begin{pmatrix}
Line 290: Line 278:
 
the $\BLUE$ to be equal (with probability $1$).
 
the $\BLUE$ to be equal (with probability $1$).
  
'''Theorem 3'''.[$\OLSE$ vs. $\BLUE$]\label{cor: olseblue}Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}.$
+
'''Theorem 3'''.[$\OLSE$ vs. $\BLUE$] ''Consider the general linear model'' $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$.
Then $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ if and only
+
''Then'' $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ ''if and only if any one of the following six equivalent conditions holds. (Note:'' $\mx{V}$ ''may be replaced by its Moore--Penrose inverse''
if any one of the following six equivalent conditions holds.
+
$\mx{V}^+$ ''and'' $\mx{H}$ ''and'' $\mx{M} = \mx I_n - \mx H$ ''may be interchanged.)''
$($Note: $\mx{V}$ may be replaced by its Moore--Penrose inverse
+
:(1) $\mx{HV} = \mx{VH} $,  
$\mx{V}^+$ and $\mx{H}$ and $\mx{M} = \mx I_n - \mx H$ may be
+
:(2) $ \mx{H}\mx{V}\mx{M} = \mx{0} $,  
interchanged.$)$
+
:(3) $ \C(\mx{V}\mx{H})\subset\C(\mx H) $,
\begin{align*}
+
:(4) $ \C(\mx{X}) $ ''has a basis comprising'' $ r= \rank(\mx X) $ ''orthonormal eigenvectors of'' $ \mx V $,  
&{ {\rm (1)}} \; \mx{HV} = \mx{VH} ,  
+
:(5) $ \mx{V} = \mx{HAH} + \mx{MBM} $  ''for some'' $ \mx A $ ''and'' $ \mx B $,  
 
+
:(6) $ \mx{V} = \alpha\mx{I}_n + \mx{HKH} + \mx{M}\mx L\mx M $ ''for some'' $ \alpha \in \rz $, ''and'' $ \mx K $ ''and'' $\mx L$.  
&{ {\rm (2)}} \; \mx{H}\mx{V}\mx{M} = \mx{0} \,,  
 
 
 
&{ {\rm (3)}} \; \C(\mx{V}\mx{H})\subset\C(\mx H) ,
 
 
 
 
 
&{ {\rm (4)}} \; \C(\mx{X}) \text{ has a basis comprising
 
} r= \rank(\mx X) \text{  orthonormal eigenvectors of } \mx V,  
 
 
 
 
 
&{ {\rm (5)}}\; \mx{V} = \mx{HAH} + \mx{MBM}
 
\text{ for some } \mx A \text{  and } \mx B,  
 
 
 
 
 
&{ {\rm (6)}}\; \mx{V} = \alpha\mx{I}_n +
 
\mx{HKH} + \mx{M}\mx L\mx M \text{ for some } \alpha \in \rz \text{ , and } \mx K \text{  and
 
} \mx L.
 
\end{align*}
 
.
 
  
 
Theorem 3 shows at once that
 
Theorem 3 shows at once that
Line 326: Line 296:
 
Consider now two linear models
 
Consider now two linear models
 
$ \M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \}$
 
$ \M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \}$
and $
+
and $ \M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \} $,
\; \M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$
 
 
which differ only in their covariance matrices.
 
which differ only in their covariance matrices.
For the proof of
+
For the proof of the
the
 
 
following proposition and related discussion, see, e.g.,
 
following proposition and related discussion, see, e.g.,
 
Rao (1971, Th. 5.2, Th. 5.5),
 
Rao (1971, Th. 5.2, Th. 5.5),
and
+
and Mitra and Moore (1973, Th. 3.3, Th. 4.1--4.2).
Mitra and Moore (1973, Th. 3.3, Th. 4.1--4.2).
 
  
'''Theorem 4'''. \label{propo: 1111}
+
'''Theorem 4'''.
Consider the linear models
+
''Consider the linear models''
 
$ \M_1 = \{\mx y, \, \mx X\BETA, \, \mx V_1 \}$
 
$ \M_1 = \{\mx y, \, \mx X\BETA, \, \mx V_1 \}$
and
+
''and''
 
$ \M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$
 
$ \M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$
and let the notation
+
''and let the notation''
$ \{\BLUE(\mx X\BETA \mid \M_1) \} \subset
+
$ \{\BLUE(\mx X\BETA \mid \M_1) \} \subset \{\BLUE(\mx X\BETA \mid \M_2) \} $
\{\BLUE(\mx X\BETA \mid \M_2) \} $
+
''mean that every representation of the'' $\BLUE$ ''for'' $\mx X\BETA$ ''under'' $\M_1$
mean that every representation of the $\BLUE$ for $\mx X\BETA$ under $\M_1$
+
''remains the'' $\BLUE$ ''for'' $\mx X\BETA$ ''under'' $\M_2$. ''Then the following statements are equivalent:''
remains the $\BLUE$ for $\mx X\BETA$ under $\M_2$. Then the following
+
:(1) $ \{ \BLUE(\mx X\BETA \mid \M_1) \} \subset
statements are equivalent:
+
\{ \BLUE(\mx X\BETA \mid \M_2) \} $,
\begin{align*}
+
:(2) $ \C(\mx V_2\mx X^{\bot}) \subset \C(\mx V_1 \mx X^\bot) $,
&{ {\rm (1)}} \;
+
:(3) $ \mx V_2 = \mx V_1+ \mx{X} \mx N_1 \mx X' +
\{ \BLUE(\mx X\BETA \mid \M_1) \} \subset
+
\mx V_1\mx M\mx N_2 \mx M \mx V_1 $, ''for some'' $ \mx N_1 $ ''and'' $ \mx N_2 $,   
\{ \BLUE(\mx X\BETA \mid \M_2) \} ,
+
:(4) $ \mx V_2 =
 
+
\mx{X} \mx N_3 \mx X' + \mx V_1\mx M\mx N_4 \mx M \mx V_1 $, ''for some'' $ \mx N_3 $ ''and'' $ \mx N_4 $.
&{ {\rm (2)}} \;
 
\C(\mx V_2\mx X^{\bot}) \subset \C(\mx V_1 \mx X^\bot) ,
 
 
 
&{ {\rm (3)}} \;
 
\mx V_2 = \mx V_1+ \mx{X} \mx N_1 \mx X' +
 
\mx V_1\mx M\mx N_2 \mx M \mx V_1,
 
\text{ for some } \mx N_1 \text{  and } \mx N_2,   
 
 
 
&{ {\rm (4)}} \;
 
\mx V_2 =
 
\mx{X} \mx N_3 \mx X' + \mx V_1\mx M\mx N_4 \mx M \mx V_1,
 
\text{ for some } \mx N_3 \text{  and } \mx N_4.
 
\end{align*}
 
 
 
.
 
  
 
Notice that obviously
 
Notice that obviously
Line 400: Line 352:
 
\begin{equation*}
 
\begin{equation*}
 
\E\begin{pmatrix}
 
\E\begin{pmatrix}
\mx y  
+
\mx y \\
 
 
 
\mx y_f
 
\mx y_f
 
\end{pmatrix} =
 
\end{pmatrix} =
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx X\BETA  
+
\mx X\BETA \\
 
 
 
\mx X_f\BETA
 
\mx X_f\BETA
 
\end{pmatrix} , \quad
 
\end{pmatrix} , \quad
 
\cov\begin{pmatrix}
 
\cov\begin{pmatrix}
\mx y  
+
\mx y \\
 
 
 
\mx y_f
 
\mx y_f
 
\end{pmatrix}
 
\end{pmatrix}
 
=
 
=
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx V = \mx V_{11} & \mx{V}_{12}  
+
\mx V = \mx V_{11} & \mx{V}_{12} \\
 
 
 
\mx{V}_{21} & \mx V_{22}
 
\mx{V}_{21} & \mx V_{22}
 
\end{pmatrix} ,
 
\end{pmatrix} ,
Line 425: Line 373:
 
\M_f = \left \{
 
\M_f = \left \{
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx y  
+
\mx y \\
 
 
 
\mx y_f
 
\mx y_f
 
\end{pmatrix},\,
 
\end{pmatrix},\,
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx X\BETA  
+
\mx X\BETA \\
 
 
 
\mx X _f\BETA
 
\mx X _f\BETA
 
\end{pmatrix},\,
 
\end{pmatrix},\,
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx V & \mx{V}_{12}  
+
\mx V & \mx{V}_{12} \\
 
 
 
\mx{V}_{21} & \mx V_{22}
 
\mx{V}_{21} & \mx V_{22}
 
\end{pmatrix} \right \}.
 
\end{pmatrix} \right \}.
Line 448: Line 393:
 
Now an unbiased linear predictor $\mx{Ay}$ is the
 
Now an unbiased linear predictor $\mx{Ay}$ is the
 
best linear unbiased predictor, $\BLUP$, for $\mx y_f$
 
best linear unbiased predictor, $\BLUP$, for $\mx y_f$
if the L&ouml;wner ordering
+
if the Löwner ordering
 
\begin{equation*}
 
\begin{equation*}
 
\cov(\mx{Ay}-\mx y_f) \leq_{ {\rm L}} \cov(\mx{By}-\mx y_f)
 
\cov(\mx{Ay}-\mx y_f) \leq_{ {\rm L}} \cov(\mx{By}-\mx y_f)
Line 461: Line 406:
 
Isotalo and Puntanen (2006, p. 1015).
 
Isotalo and Puntanen (2006, p. 1015).
  
'''Theorem 5'''.[Fundamental $\BLUP$ equation]\label{propo:BLUP-funda}
+
'''Theorem 5''' (Fundamental $\BLUP$ equation)
Consider the linear model
+
''Consider the linear model''
$\M_f,$ where
+
$\M_f$, ''where''
$\mx{X}_f\BETA$ is a given estimable parametric function.
+
$\mx{X}_f\BETA$ ''is a given estimable parametric function. Then the linear estimator'' $\mx{Ay}$
Then the linear estimator $\mx{Ay}$ is the best
+
''is the best linear unbiased predictor'' ($\BLUP$) ''for'' $\mx y_f$
linear unbiased predictor { {\rm (}}$\BLUP${ {\rm )}} for $\mx y_f$ if
+
''if and only if'' $\mx{A}$ ''satisfies the equation''
and only if $\mx{A}$ satisfies the equation
 
 
\begin{equation*}
 
\begin{equation*}
 
\mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ).
 
\mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ).
 
\end{equation*}
 
\end{equation*}
In terms of Pandora's Box (Theorem 2), $\mx{Ay}$ is the $\BLUP$
+
''In terms of Pandora's Box (Theorem 2),'' $\mx{Ay}$ ''is the'' $\BLUP$
for $\mx y_f$ if and only if there exists a matrix~$\mx L$ such that
+
''for'' $\mx y_f$ ''if and only if there exists a matrix'' $\mx L$ ''such that''
$\mx{A}$ satisfies the equation
+
$\mx{A}$ ''satisfies the equation''
 
\begin{equation*}
 
\begin{equation*}
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx V & \mx X  
+
\mx V & \mx X \\
 
 
 
\mx X' & \mx 0
 
\mx X' & \mx 0
 
\end{pmatrix}
 
\end{pmatrix}
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx A'  
+
\mx A' \\
 
 
 
\mx L
 
\mx L
 
\end{pmatrix} =
 
\end{pmatrix} =
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx{V}_{12}  
+
\mx{V}_{12} \\
 
 
 
\mx X_{f}'
 
\mx X_{f}'
 
\end{pmatrix}.
 
\end{pmatrix}.
 
\end{equation*}
 
\end{equation*}
.
 
  
 
== The Mixed Model ==
 
== The Mixed Model ==
Line 515: Line 455:
  
 
'''Theorem 6'''.
 
'''Theorem 6'''.
Consider the mixed model
+
''Consider the mixed model''
 
$ \M_{\mathrm{mix}}
 
$ \M_{\mathrm{mix}}
 
= \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \}.$
 
= \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \}.$
Then the linear estimator
+
''Then the linear estimator''
$\mx B \mx y$ is the $\BLUE$ for $\mx X\BETA$ if and only if
+
$\mx B \mx y$ ''is the'' $\BLUE$ ''for'' $\mx X\BETA$ ''if and only if''
 
\begin{equation*}
 
\begin{equation*}
 
\mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) ,
 
\mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) ,
 
\end{equation*}
 
\end{equation*}
where $\SIGMA= \mx Z\mx D\mx Z' + \mx R$.
+
''where'' $\SIGMA= \mx Z\mx D\mx Z' + \mx R$.
Moreover,
+
''Moreover,''
$\mx A \mx y$ is the $\BLUP$ for $\GAMMA$ if and only if
+
$\mx A \mx y$ ''is the'' $\BLUP$ ''for'' $\GAMMA$ ''if and only if''
 
\begin{equation*}
 
\begin{equation*}
 
\mx A(\mx X : \SIGMA \mx X^{\bot}) = (\mx 0 : \mx{D}\mx{Z}' \mx X^{\bot}).
 
\mx A(\mx X : \SIGMA \mx X^{\bot}) = (\mx 0 : \mx{D}\mx{Z}' \mx X^{\bot}).
 
\end{equation*}
 
\end{equation*}
In terms of
+
''In terms of Pandora's Box (Theorem 2),'' $\mx A \mx y = \BLUP(\GAMMA)$
Pandora's Box (Theorem 2), $\mx A \mx y = \BLUP(\GAMMA)$ if and only if there
+
''if and only if there exists a matrix'' $\mx L$ ''such that'' $\mx{A}$ ''satisfies the equation''
exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation
 
 
\begin{equation*}
 
\begin{equation*}
 
\begin{pmatrix}
 
\begin{pmatrix}
\SIGMA & \mx X  
+
\SIGMA & \mx X \\
 
 
 
\mx X' & \mx 0
 
\mx X' & \mx 0
 
\end{pmatrix}
 
\end{pmatrix}
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx A'  
+
\mx A' \\
 
 
 
\mx L
 
\mx L
 
\end{pmatrix} =
 
\end{pmatrix} =
 
\begin{pmatrix}
 
\begin{pmatrix}
\mx Z \mx D  
+
\mx Z \mx D \\
 
 
 
\mx 0
 
\mx 0
 
\end{pmatrix}.
 
\end{pmatrix}.
 
\end{equation*}
 
\end{equation*}
.
 
  
 
For the equality
 
For the equality
Line 567: Line 502:
 
{|
 
{|
 
|-
 
|-
|valign="top"|{{Ref|1}}||valign="top"|  Anderson, T. W. (1948). On the theory of testing serial correlation. ''Skandinavisk Aktuarietidskrift'', '''31''', 88--116. \
+
|valign="top"|{{Ref|1}}||valign="top"|  Anderson, T. W. (1948). On the theory of testing serial correlation. ''Skandinavisk Aktuarietidskrift'', '''31''', 88--116.
 
|-
 
|-
|valign="top"|{{Ref|2}}||valign="top"|  Baksalary, Jerzy K.; Rao, C. Radhakrishna and Markiewicz, Augustyn (1992). A study of the influence of the `natural restrictions' on estimation problems in the singular {G}auss--{M}arkov model, ''Journal of Statistical Planning and Inference'', '''31''', 335--351. \
+
|valign="top"|{{Ref|2}}||valign="top"|  Baksalary, Jerzy K.; Rao, C. Radhakrishna and Markiewicz, Augustyn (1992). A study of the influence of the `natural restrictions' on estimation problems in the singular Gauss--Markov model, ''Journal of Statistical Planning and Inference'', '''31''', 335--351.
 
|-
 
|-
|valign="top"|{{Ref|3}}||valign="top"|  Christensen, Ronald (2002). ''Plane Answers to Complex Questions: The Theory of Linear Models,'' 3rd Edition. Springer, New York. \
+
|valign="top"|{{Ref|3}}||valign="top"|  Christensen, Ronald (2002). ''Plane Answers to Complex Questions: The Theory of Linear Models,'' 3rd Edition. Springer, New York.
 
|-
 
|-
|valign="top"|{{Ref|4}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010a). Effect of adding regressors on the equality of the BLUEs under two linear models. ''Journal of Statistical Planning and Inference'', '''140''', 104--110, \
+
|valign="top"|{{Ref|4}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010a). Effect of adding regressors on the equality of the BLUEs under two linear models. ''Journal of Statistical Planning and Inference'', '''140''', 104--110,
 
|-
 
|-
|valign="top"|{{Ref|5}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010b). Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. ''Statistical Papers'', '''51''', 465--475. \
+
|valign="top"|{{Ref|5}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010b). Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. ''Statistical Papers'', '''51''', 465--475.
 
|-
 
|-
|valign="top"|{{Ref|6}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010c). On the equality of the BLUPs under two linear mixed models. ''Metrika'', available online, DOI 10.1007/s00184-010-0308-6. \
+
|valign="top"|{{Ref|6}}||valign="top"|  Haslett, Stephen J. and Puntanen, Simo (2010c). On the equality of the BLUPs under two linear mixed models. ''Metrika'', available online, DOI 10.1007/s00184-010-0308-6.
 
|-
 
|-
|valign="top"|{{Ref|7}}||valign="top"|  Isotalo, Jarkko and Puntanen, Simo (2006). Linear prediction sufficiency for new observations in the general Gauss--Markov model. ''Communications in Statistics: Theory and Methods'', '''35''', 1011--1023. \
+
|valign="top"|{{Ref|7}}||valign="top"|  Isotalo, Jarkko and Puntanen, Simo (2006). Linear prediction sufficiency for new observations in the general Gauss--Markov model. ''Communications in Statistics: Theory and Methods'', '''35''', 1011--1023.
 
|-
 
|-
|valign="top"|{{Ref|8}}||valign="top"|  Kruskal, William (1967). When are Gauss--Markov and least squares estimators identical? {A} coordinate-free approach. ''The Annals of Mathematical Statistics'', '''39''', 70--75. \
+
|valign="top"|{{Ref|8}}||valign="top"|  Kruskal, William (1967). When are Gauss--Markov and least squares estimators identical? {A} coordinate-free approach. ''The Annals of Mathematical Statistics'', '''39''', 70--75.
 
|-
 
|-
|valign="top"|{{Ref|9}}||valign="top"|  Mitra, Sujit Kumar and Moore, Betty Jeanne (1973). Gauss--Markov estimation with an incorrect dispersion matrix. ''Sankhya, Series~A'', '''35''', 139--152. \
+
|valign="top"|{{Ref|9}}||valign="top"|  Mitra, Sujit Kumar and Moore, Betty Jeanne (1973). Gauss--Markov estimation with an incorrect dispersion matrix. ''Sankhya, Series A'', '''35''', 139--152.
 
|-
 
|-
|valign="top"|{{Ref|10}}||valign="top"|  Puntanen, Simo and Styan, George P. H. (1989). The equality of the ordinary least squares estimator and the best linear unbiased estimator [with comments by Oscar Kempthorne and by Shayle~R. Searle and with ``Reply'' by the authors]. ''The American Statistician'', '''43''', 153--164. \
+
|valign="top"|{{Ref|10}}||valign="top"|  Puntanen, Simo and Styan, George P. H. (1989). The equality of the ordinary least squares estimator and the best linear unbiased estimator [with comments by Oscar Kempthorne and by Shayle R. Searle and with "Reply" by the authors]. ''The American Statistician'', '''43''', 153--164.
 
|-
 
|-
|valign="top"|{{Ref|11}}||valign="top"|  Puntanen, Simo; Styan, George P. H. and Werner, Hans Joachim (2000). Two matrix-based proofs that the linear estimator ''Gy'' is the best linear unbiased estimator. ''Journal of Statistical Planning and Inference'', '''88''', 173--179. \
+
|valign="top"|{{Ref|11}}||valign="top"|  Puntanen, Simo; Styan, George P. H. and Werner, Hans Joachim (2000). Two matrix-based proofs that the linear estimator ''Gy'' is the best linear unbiased estimator. ''Journal of Statistical Planning and Inference'', '''88''', 173--179.
 
|-
 
|-
|valign="top"|{{Ref|12}}||valign="top"|  Rao, C. Radhakrishna (1967). Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In ''Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, 1965/1966'', vol. 1 (Eds. Lucien~M. Le Cam & Jerzy Neyman), University of California Press, Berkeley, pp. 355--372. \
+
|valign="top"|{{Ref|12}}||valign="top"|  Rao, C. Radhakrishna (1967). Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In ''Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, 1965/1966'', vol. 1 (Eds. Lucien M. Le Cam & Jerzy Neyman), University of California Press, Berkeley, pp. 355--372.
 
|-
 
|-
|valign="top"|{{Ref|13}}||valign="top"|  Rao, C. Radhakrishna (1971). Unified theory of linear estimation. ''Sankhya, Series~A'', '''33''', 371--394. [Corrigenda (1972), '''34''', p. 194 and p. 477.] \
+
|valign="top"|{{Ref|13}}||valign="top"|  Rao, C. Radhakrishna (1971). Unified theory of linear estimation. ''Sankhya, Series A'', '''33''', 371--394. [Corrigenda (1972), '''34''', p. 194 and p. 477.]
 
|-
 
|-
|valign="top"|{{Ref|14}}||valign="top"|  Rao, C. Radhakrishna (1974). Projectors, generalized inverses and the BLUE's. ''Journal of the Royal Statistical Society, Series~B'', '''36''', 442--448. \
+
|valign="top"|{{Ref|14}}||valign="top"|  Rao, C. Radhakrishna (1974). Projectors, generalized inverses and the BLUE's. ''Journal of the Royal Statistical Society, Series B'', '''36''', 442--448.
 
|-
 
|-
|valign="top"|{{Ref|15}}||valign="top"|  Watson, Geoffrey S. (1967). Linear least squares regression. \emph {The Annals of Mathematical Statistics}, '''38''', 1679--1699. \
+
|valign="top"|{{Ref|15}}||valign="top"|  Watson, Geoffrey S. (1967). Linear least squares regression. ''The Annals of Mathematical Statistics'', '''38''', 1679--1699.
 
|-
 
|-
|valign="top"|{{Ref|16}}||valign="top"|  Zyskind, George (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. ''The Annals of Mathematical Statistics'', '''38''', 1092--1109. \
+
|valign="top"|{{Ref|16}}||valign="top"|  Zyskind, George (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. ''The Annals of Mathematical Statistics'', '''38''', 1092--1109.
 
|-
 
|-
 
|valign="top"|{{Ref|17}}||valign="top"|  Zyskind, George and Martin, Frank B. (1969). On best linear estimation and general Gauss--Markov theorem in linear models with arbitrary nonnegative covariance structure. ''SIAM Journal on Applied Mathematics'', '''17''', 1190--1202.
 
|valign="top"|{{Ref|17}}||valign="top"|  Zyskind, George and Martin, Frank B. (1969). On best linear estimation and general Gauss--Markov theorem in linear models with arbitrary nonnegative covariance structure. ''SIAM Journal on Applied Mathematics'', '''17''', 1190--1202.
|-
 
 
|}
 
|}
  

Latest revision as of 16:20, 10 March 2024

Copyright notice
This article Best linear unbiased estimation in linear models was adapted from an original article by Simo Puntanen, George P.H. Styan, which appeared in StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies. The original article ([http://statprob.com/encyclopedia/BestLinearUnbiasedEstimatinInLinearModels.html StatProb Source], Local Files: pdf | tex) is copyrighted by the author(s), the article has been donated to Encyclopedia of Mathematics, and its further issues are under Creative Commons Attribution Share-Alike License'. All pages from StatProb are contained in the Category StatProb.

$\def\mx#1{ {\mathbf{#1}}}$ $\def\BETA{\beta}\def\BETAH{ {\hat\beta}}\def\BETAT{ {\tilde\beta}}\def\betat{\tilde\beta}$ $\def\C{ {\mathscr C}}$ $\def\cov{\mathrm{cov}}\def\M{ {\mathscr M}}$ $\def\NS{ {\mathscr N}}\def\OLSE{ {\small\mathrm{OLSE}}}$ $\def\rank{ {\rm rank}} \def\tr{ { \rm trace}}$ $\def\rz{ {\mathbf{R}}} \def\SIGMA{\Sigma} \def\var{ {\rm var}}$ $\def\BLUE}{\small\mathrm{BLUE}}$ $\def\BLUP}{\small\mathrm{BLUP}}$ $\def\EPS{\varepsilon}$ $\def\EE{E}$ $\def\E{E}$ $\def\GAMMA{\gamma}$ 2020 Mathematics Subject Classification: Primary: 62J05 [MSN][ZBL]

Best Linear Unbiased Estimation in Linear Models
Simo Puntanen [1]


University of Tampere, Finland

and

George P. H. Styan [2]


McGill University, Montréal, Canada

Keywords and Phrases: Best linear unbiased, BLUE, BLUP, Gauss--Markov Theorem, Generalized inverse, Ordinary least squares, OLSE.

Introduction

In this article we consider the general linear model (Gauss--Markov model) \begin{equation*} \mx y = \mx X \BETA + \EPS, \quad \text{or in short } \M = \{ \mx y, \, \mx X \BETA, \, \sigma^2 \mx V \}, \end{equation*} where $\mx X$ is a known $n\times p$ model matrix, the vector $\mx y$ is an observable $n$-dimensional random vector, $\BETA$ is a $p\times 1$ vector of unknown parameters, and $\EPS$ is an unobservable vector of random errors with expectation $\EE(\EPS ) = \mx 0,$ and covariance matrix $\cov( \EPS) = \sigma^2 \mx V,$ where $\sigma^2 >0$ is an unknown constant. The nonnegative definite (possibly singular) matrix $\mx V $ is known. In our considerations $\sigma ^2$ has no role and hence we may put $\sigma^2=1.$

As regards the notation, we will use the symbols $\mx A',$ $\mx A^{-},$ $\mx A^{+},$ $\C(\mx A),$ $\C(\mx A)^{\bot},$ and $\NS(\mx A)$ to denote, respectively, the transpose, a generalized inverse, the Moore--Penrose inverse, the column space, the orthogonal complement of the column space, and the null space, of the matrix $\mx A$. By $(\mx A:\mx B)$ we denote the partitioned matrix with $\mx A$ and $\mx B$ as submatrices. By $\mx A^{\bot}$ we denote any matrix satisfying $\C(\mx A^{\bot}) = \NS(\mx A') = \C(\mx A)^{\bot}.$ Furthermore, we will write $\mx P_{\mx A} = \mx A\mx A^{+} = \mx A(\mx A'\mx A)^{-}\mx A'$ to denote the orthogonal projector (with respect to the standard inner product) onto $\C(\mx A).$ In particular, we denote $\mx{H} = \mx P_{\mx X}$ and $ \mx{M} = \mx I_n - \mx H$. One choice for $\mx X^{\bot}$ is of course the projector $\mx M$.

Let $\mx K' \BETA$ be a given vector of parametric functions specified by $\mx K' \in \rz^{q\times p}.$ Our object is to find a (homogeneous) linear estimator $\mx A \mx y$ which would provide an unbiased and in some sense "best" estimator for $\mx K' \BETA$ under the model $\M.$ However, not all parametric functions have linear unbiased estimators; those which have are called estimable parametric functions, and then there exists a matrix $\mx A$ such that \begin{equation*} \E(\mx{Ay}) = \mx{AX}\BETA = \mx K' \BETA \quad \text{for all } \BETA \in \rz^p. \end{equation*} Hence $\mx{K}' \BETA$ is estimable if and only if there exists a matrix $\mx A$ such that $\mx{K}' = \mx{A}\mx{X}$, i.e., $ \C(\mx K ) \subset \C(\mx X')$.

The ordinary least squares estimator of $\mx K' \BETA$ is defined as $ \OLSE(\mx K' \BETA) = \mx K' \BETAH, $ where $\BETAH$ is any solution to the normal equation $\mx X' \mx X \BETAH = \mx X' \mx y$; hence $\BETA = \BETAH$ minimizes $(\mx y - \mx X\BETA)' (\mx y - \mx X\BETA)$ and it can be expressed as $\BETAH = (\mx X' \mx X) ^{-}\mx X' \mx y,$ while $\mx X\BETAH = \mx H \mx y.$ Now the condition $\C(\mx K ) \subset \C(\mx X')$ guarantees that $\mx K'\BETAH$ is unique, even though $\BETAH$ may not be unique.

The Best Linear Unbiased Estimator (BLUE)

The expectation $\mx X\BETA$ is trivially estimable and $\mx{Gy}$ is unbiased for $\mx X\BETA$ whenever $ \mx{G}\mx X = \mx{X}.$ An unbiased linear estimator $\mx{Gy}$ for $\mx X\BETA$ is defined to be the best linear unbiased estimator, $\BLUE$, for $\mx X\BETA$ under $\M$ if \begin{equation*} \cov( \mx{G} \mx y) \leq_{ {\rm L}} \cov( \mx{L} \mx y) \quad \text{for all } \mx{L} \colon \mx{L}\mx X = \mx{X}, \end{equation*} where "$\leq_\text{L}$" refers to the Löwner partial ordering. In other words, $\mx{G} \mx y$ has the smallest covariance matrix (in the Löwner sense) among all linear unbiased estimators. We denote the $\BLUE$ of $\mx X\BETA$ as $ \BLUE(\mx X\BETA) = \mx X \BETAT. $ If $\mx X$ has full column rank, then $\BETA$ is estimable and an unbiased estimator $\mx A\mx y$ is the $\BLUE$ for $\BETA$ if $ \mx{AVA}' \leq_{ {\rm L}} \mx{BVB}' $ for all $\mx{B}$ such that $ \mx{BX} = \mx{I}_p. $ The Löwner ordering is a very strong ordering implying for example \begin{gather*} \var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p , \tr [\cov(\BETAT)] \le \tr [\cov(\BETA^{*})] , \qquad \det[\cov(\BETAT)] \le \det[\cov(\BETA^{*})], \end{gather*} for any linear unbiased estimator $\BETA^{*}$ of $\BETA$; here $\var$ refers to the variance and "det" denotes the determinant.

The following theorem gives the "Fundamental $\BLUE$ equation"; see, e.g., Rao (1967), Zyskind (1967) and Puntanen, Styan and Werner (2000).

Theorem 1. Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if $\mx G$ satisfies the equation $$ \label{eq: 30jan09-fundablue} \mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}). \tag{1}$$ The corresponding condition for $\mx{Ay}$ to be the $\BLUE$ of an estimable parametric function $\mx{K}' \BETA$ is $ \mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0})$.

It is sometimes convenient to express (1) in the following form, see Rao (1971).

Theorem 2.[Pandora's Box] Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then the estimator $\mx{Gy}$ is the $\BLUE$ for $\mx X\BETA$ if and only if there exists a matrix $\mx{L} \in \rz^{p \times n}$ so that $\mx G$ is a solution to \begin{equation*} \begin{pmatrix} \mx V & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx G' \\ \mx{L} \end{pmatrix} = \begin{pmatrix} \mx 0 \\ \mx X' \end{pmatrix}. \end{equation*}

The equation (1) has a unique solution for $\mx G$ if and only if $\C(\mx X : \mx V) = \rz^n.$ Notice that under $\M$ we assume that the observed value of $\mx y$ belongs to the subspace $\C(\mx X : \mx V)$ with probability $1$; this is the consistency condition of the linear model, see, e.g., Baksalary, Rao and Markiewicz (1992). The consistency condition means, for example, that whenever we have some statements which involve the random vector $\mx y$, these statements need hold only for those values of $\mx y$ that belong to $\C(\mx{X}:\mx{V}).$ The general solution for $\mx G$ can be expressed, for example, in the following ways: \[ \mx G_1 = \mx{X}(\mx{X}'\mx{W}^{-}\mx{X})^{-}\mx{X}'\mx{W}^{-} + \mx F_{1}(\mx{I }_n - \mx W\mx W^{-} ) , \] \[ \mx G_2 = \mx{H} - \mx{HVM}(\mx{MVM})^{-}\mx{M} + \mx F_{2}[\mx{I}_n - \mx{MVM}( \mx{MVM} )^{-} ]\mx M , \] where $\mx F_{1}$ and $\mx F_{2}$ are arbitrary matrices, $\mx {W}= \mx V + \mx X\mx U\mx X'$ and $\mx U$ is any arbitrary conformable matrix such that $\C(\mx W) = \C(\mx X : \mx V).$ Notice that even though $\mx G$ may not be unique, the numerical value of $\mx G\mx y$ is unique because $\mx y \in \C(\mx X : \mx V).$ If $\mx V$ is positive definite, then $\BLUE(\mx X\BETA) = \mx X(\mx X' \mx V^{-1} \mx X)^{-} \mx X' \mx V^{-1} \mx y.$ Clearly $\OLSE(\mx X\BETA) = \mx H\mx y$ is the $\BLUE$ under $\{ \mx y, \, \mx X\BETA , \, \sigma^2\mx I \}.$ It is also worth noting that the matrix $\mx G$ satisfying (1) can be interpreted as a projector: it is a projector onto $\C(\mx X)$ along $\C(\mx V\mx X^{\bot}),$ see Rao (1974).

OLSE vs. BLUE

Characterizing the equality of the Ordinary Least Squares Estimator $(\OLSE)$ and the $\BLUE$ has received a lot of attention in the literature, since Anderson (1948), but the major breakthroughs were made by Rao (1967) and Zyskind (1967); for a detailed review, see Puntanen and Styan (1989). For some further references from those years we may mention Kruskal (1968), Watson (1967), and Zyskind and Martin (1969).

We present below six characterizations for the $\OLSE$ and the $\BLUE$ to be equal (with probability $1$).

Theorem 3.[$\OLSE$ vs. $\BLUE$] Consider the general linear model $ \M =\{\mx y,\,\mx X\BETA,\,\mx V\}$. Then $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ if and only if any one of the following six equivalent conditions holds. (Note: $\mx{V}$ may be replaced by its Moore--Penrose inverse $\mx{V}^+$ and $\mx{H}$ and $\mx{M} = \mx I_n - \mx H$ may be interchanged.)

(1) $\mx{HV} = \mx{VH} $,
(2) $ \mx{H}\mx{V}\mx{M} = \mx{0} $,
(3) $ \C(\mx{V}\mx{H})\subset\C(\mx H) $,
(4) $ \C(\mx{X}) $ has a basis comprising $ r= \rank(\mx X) $ orthonormal eigenvectors of $ \mx V $,
(5) $ \mx{V} = \mx{HAH} + \mx{MBM} $ for some $ \mx A $ and $ \mx B $,
(6) $ \mx{V} = \alpha\mx{I}_n + \mx{HKH} + \mx{M}\mx L\mx M $ for some $ \alpha \in \rz $, and $ \mx K $ and $\mx L$.

Theorem 3 shows at once that under $\{ \mx y, \, \mx X\BETA, \, \mx I_n \}$ the $\OLSE$ of $\mx X\BETA$ is trivially the $\BLUE$; this result is often called the Gauss--Markov Theorem.

Two Linear Models

Consider now two linear models $ \M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $ \M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \} $, which differ only in their covariance matrices. For the proof of the following proposition and related discussion, see, e.g., Rao (1971, Th. 5.2, Th. 5.5), and Mitra and Moore (1973, Th. 3.3, Th. 4.1--4.2).

Theorem 4. Consider the linear models $ \M_1 = \{\mx y, \, \mx X\BETA, \, \mx V_1 \}$ and $ \M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \},$ and let the notation $ \{\BLUE(\mx X\BETA \mid \M_1) \} \subset \{\BLUE(\mx X\BETA \mid \M_2) \} $ mean that every representation of the $\BLUE$ for $\mx X\BETA$ under $\M_1$ remains the $\BLUE$ for $\mx X\BETA$ under $\M_2$. Then the following statements are equivalent:

(1) $ \{ \BLUE(\mx X\BETA \mid \M_1) \} \subset \{ \BLUE(\mx X\BETA \mid \M_2) \} $,
(2) $ \C(\mx V_2\mx X^{\bot}) \subset \C(\mx V_1 \mx X^\bot) $,
(3) $ \mx V_2 = \mx V_1+ \mx{X} \mx N_1 \mx X' + \mx V_1\mx M\mx N_2 \mx M \mx V_1 $, for some $ \mx N_1 $ and $ \mx N_2 $,
(4) $ \mx V_2 = \mx{X} \mx N_3 \mx X' + \mx V_1\mx M\mx N_4 \mx M \mx V_1 $, for some $ \mx N_3 $ and $ \mx N_4 $.

Notice that obviously \begin{align*} \{ \BLUE(\mx X \BETA \mid \M_1) \} = \{ \BLUE(\mx X \BETA \mid \M_2) \} \iff \C(\mx V_2\mx X^{\bot}) = \C(\mx V_1 \mx X^\bot). \end{align*} For the equality between the $\BLUE$s of $\mx X_1\BETA_1$ under two partitioned models, see Haslett and Puntanen (2010a).

Model with New Observations: Best Linear Unbiased Predictor (BLUP)

Consider the model $ \M = \{\mx y,\,\mx X\BETA,\,\mx V\},$ and let $\mx y_f$ denote an $m\times 1$ unobservable random vector containing new observations. The new observations are assumed to follow the linear model $ \mx y_f = \mx X_f\BETA +\EPS_f ,$ where $\mx X_f$ is a known $m\times p$ model matrix associated with new observations, $\BETA$ is the same vector of unknown parameters as in $\M$, and $\EPS_f$ is an $m \times 1$ random error vector associated with new observations. Our goal is to predict the random vector $\mx y_f$ on the basis of $\mx y$. The expectation and the covariance matrix are \begin{equation*} \E\begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix} = \begin{pmatrix} \mx X\BETA \\ \mx X_f\BETA \end{pmatrix} , \quad \cov\begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix} = \begin{pmatrix} \mx V = \mx V_{11} & \mx{V}_{12} \\ \mx{V}_{21} & \mx V_{22} \end{pmatrix} , \end{equation*} which we may write as \begin{equation*} \M_f = \left \{ \begin{pmatrix} \mx y \\ \mx y_f \end{pmatrix},\, \begin{pmatrix} \mx X\BETA \\ \mx X _f\BETA \end{pmatrix},\, \begin{pmatrix} \mx V & \mx{V}_{12} \\ \mx{V}_{21} & \mx V_{22} \end{pmatrix} \right \}. \end{equation*}

A linear predictor $\mx{Ay}$ is said to be unbiased for $\mx y_f$ if $ \E(\mx{Ay}) = \E(\mx{y}_f) = \mx X_f\BETA$ for all $\BETA\in\rz^{p}.$ Then the random vector $\mx y_f$ is said to be unbiasedly predictable. Now an unbiased linear predictor $\mx{Ay}$ is the best linear unbiased predictor, $\BLUP$, for $\mx y_f$ if the Löwner ordering \begin{equation*} \cov(\mx{Ay}-\mx y_f) \leq_{ {\rm L}} \cov(\mx{By}-\mx y_f) \end{equation*} holds for all $\mx B$ such that $\mx{By}$ is an unbiased linear predictor for $\mx{y}_f$.

The following theorem characterizes the $\BLUP$; see, e.g., Christensen (2002, p. 283), and Isotalo and Puntanen (2006, p. 1015).

Theorem 5 (Fundamental $\BLUP$ equation) Consider the linear model $\M_f$, where $\mx{X}_f\BETA$ is a given estimable parametric function. Then the linear estimator $\mx{Ay}$ is the best linear unbiased predictor ($\BLUP$) for $\mx y_f$ if and only if $\mx{A}$ satisfies the equation \begin{equation*} \mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx{Ay}$ is the $\BLUP$ for $\mx y_f$ if and only if there exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \mx V & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \\ \mx L \end{pmatrix} = \begin{pmatrix} \mx{V}_{12} \\ \mx X_{f}' \end{pmatrix}. \end{equation*}

The Mixed Model

A mixed linear model can be presented as \begin{equation*} \mx y = \mx X\BETA + \mx Z \GAMMA +\EPS , \quad \text{or shortly } \quad \M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \} , \end{equation*} where $\mx X \in \rz^{n \times p}$ and $\mx Z \in \rz^{n \times q}$ are known matrices, $\BETA \in \rz^{p}$ is a vector of unknown fixed effects, $\GAMMA$ is an unobservable vector ($q$ elements) of random effects with $\cov(\GAMMA,\EPS) = \mx 0_{q \times p}$ and \begin{equation*} \E(\GAMMA) = \mx 0_q , \quad \cov(\GAMMA) = \mx D_{q \times q}, \quad \E(\EPS) = \mx 0_n \,, \quad \cov(\EPS) = \mx R_{n\times n}. \end{equation*} This leads directly to:

Theorem 6. Consider the mixed model $ \M_{\mathrm{mix}} = \{ \mx y,\, \mx X\BETA + \mx Z\GAMMA, \, \mx D,\,\mx R \}.$ Then the linear estimator $\mx B \mx y$ is the $\BLUE$ for $\mx X\BETA$ if and only if \begin{equation*} \mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) , \end{equation*} where $\SIGMA= \mx Z\mx D\mx Z' + \mx R$. Moreover, $\mx A \mx y$ is the $\BLUP$ for $\GAMMA$ if and only if \begin{equation*} \mx A(\mx X : \SIGMA \mx X^{\bot}) = (\mx 0 : \mx{D}\mx{Z}' \mx X^{\bot}). \end{equation*} In terms of Pandora's Box (Theorem 2), $\mx A \mx y = \BLUP(\GAMMA)$ if and only if there exists a matrix $\mx L$ such that $\mx{A}$ satisfies the equation \begin{equation*} \begin{pmatrix} \SIGMA & \mx X \\ \mx X' & \mx 0 \end{pmatrix} \begin{pmatrix} \mx A' \\ \mx L \end{pmatrix} = \begin{pmatrix} \mx Z \mx D \\ \mx 0 \end{pmatrix}. \end{equation*}

For the equality between the $\BLUP$s under two mixed models, see Haslett and Puntanen (2010b, 2010c).

Note

Reprinted with permission from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science+Business Media, LLC.


References

[1] Anderson, T. W. (1948). On the theory of testing serial correlation. Skandinavisk Aktuarietidskrift, 31, 88--116.
[2] Baksalary, Jerzy K.; Rao, C. Radhakrishna and Markiewicz, Augustyn (1992). A study of the influence of the `natural restrictions' on estimation problems in the singular Gauss--Markov model, Journal of Statistical Planning and Inference, 31, 335--351.
[3] Christensen, Ronald (2002). Plane Answers to Complex Questions: The Theory of Linear Models, 3rd Edition. Springer, New York.
[4] Haslett, Stephen J. and Puntanen, Simo (2010a). Effect of adding regressors on the equality of the BLUEs under two linear models. Journal of Statistical Planning and Inference, 140, 104--110,
[5] Haslett, Stephen J. and Puntanen, Simo (2010b). Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Statistical Papers, 51, 465--475.
[6] Haslett, Stephen J. and Puntanen, Simo (2010c). On the equality of the BLUPs under two linear mixed models. Metrika, available online, DOI 10.1007/s00184-010-0308-6.
[7] Isotalo, Jarkko and Puntanen, Simo (2006). Linear prediction sufficiency for new observations in the general Gauss--Markov model. Communications in Statistics: Theory and Methods, 35, 1011--1023.
[8] Kruskal, William (1967). When are Gauss--Markov and least squares estimators identical? {A} coordinate-free approach. The Annals of Mathematical Statistics, 39, 70--75.
[9] Mitra, Sujit Kumar and Moore, Betty Jeanne (1973). Gauss--Markov estimation with an incorrect dispersion matrix. Sankhya, Series A, 35, 139--152.
[10] Puntanen, Simo and Styan, George P. H. (1989). The equality of the ordinary least squares estimator and the best linear unbiased estimator [with comments by Oscar Kempthorne and by Shayle R. Searle and with "Reply" by the authors]. The American Statistician, 43, 153--164.
[11] Puntanen, Simo; Styan, George P. H. and Werner, Hans Joachim (2000). Two matrix-based proofs that the linear estimator Gy is the best linear unbiased estimator. Journal of Statistical Planning and Inference, 88, 173--179.
[12] Rao, C. Radhakrishna (1967). Least squares theory using an estimated dispersion matrix and its application to measurement of signals. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability: Berkeley, California, 1965/1966, vol. 1 (Eds. Lucien M. Le Cam & Jerzy Neyman), University of California Press, Berkeley, pp. 355--372.
[13] Rao, C. Radhakrishna (1971). Unified theory of linear estimation. Sankhya, Series A, 33, 371--394. [Corrigenda (1972), 34, p. 194 and p. 477.]
[14] Rao, C. Radhakrishna (1974). Projectors, generalized inverses and the BLUE's. Journal of the Royal Statistical Society, Series B, 36, 442--448.
[15] Watson, Geoffrey S. (1967). Linear least squares regression. The Annals of Mathematical Statistics, 38, 1679--1699.
[16] Zyskind, George (1967). On canonical forms, non-negative covariance matrices and best and simple least squares linear estimators in linear models. The Annals of Mathematical Statistics, 38, 1092--1109.
[17] Zyskind, George and Martin, Frank B. (1969). On best linear estimation and general Gauss--Markov theorem in linear models with arbitrary nonnegative covariance structure. SIAM Journal on Applied Mathematics, 17, 1190--1202.


  1. Department of Mathematics and Statistics, FI-33014 University of Tampere, Tampere, Finland. Email: simo.puntanen@uta.fi
  2. Department of Mathematics and Statistics, McGill University, 805 ouest rue Sherbrooke Street West, Montréal (Québec), Canada H3A 2K6. Email: styan@math.mcgill.ca
How to Cite This Entry:
Best linear unbiased estimation in linear models. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Best_linear_unbiased_estimation_in_linear_models&oldid=38484