View source for M-estimator

A generalization of the maximum-likelihood estimator (MLE) in [[Mathematical statistics|mathematical statistics]] (cf. also [[Maximum-likelihood method|Maximum-likelihood method]]; [[Statistical estimator|Statistical estimator]]). Suppose one has univariate observations <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200301.png" /> which are independent and identically distributed according to a distribution <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200302.png" /> with univariate parameter <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200303.png" />. Denote by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200304.png" /> the likelihood of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200305.png" />. The maximum-likelihood estimator is defined as the value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200306.png" /> which maximizes <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200307.png" />. If <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200308.png" /> for all <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m1200309.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003010.png" />, then this is equivalent to minimizing <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003011.png" />. P.J. Huber [[#References|[a1]]] has generalized this to M-estimators, which are defined by minimizing <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003012.png" />, where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003013.png" /> is an arbitrary real function. When <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003014.png" /> has a partial derivative <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003015.png" />, then <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003016.png" /> satisfies the implicit equation

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003017.png" /></td> </tr></table>

Note that the maximum-likelihood estimator is an M-estimator, obtained by putting <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003018.png" />.

The maximum-likelihood estimator can give arbitrarily bad results when the underlying assumptions (e.g., the form of the distribution generating the data) are not satisfied (e.g., because the data contain some outliers, cf. also [[Outlier|Outlier]]). M-estimators are particularly useful in [[Robust statistics|robust statistics]], which aims to construct methods that are relatively insensitive to deviations from the standard assumptions. M-estimators with bounded <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003019.png" /> are typically robust.

Apart from the finite-sample version <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003020.png" /> of the M-estimator, there is also a functional version <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003021.png" /> defined for any [[Probability distribution|probability distribution]] <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003022.png" /> by

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003023.png" /></td> </tr></table>

Here, it is assumed that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003024.png" /> is Fisher-consistent, i.e. that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003025.png" /> for all <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003026.png" />. The influence function of a functional <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003027.png" /> in <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003028.png" /> is defined, as in [[#References|[a2]]], by

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003029.png" /></td> </tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003030.png" /> is the probability distribution which puts all its mass in the point <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003031.png" />. Therefore <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003032.png" /> describes the effect of a single outlier in <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003033.png" /> on the estimator <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003034.png" />. For an M-estimator <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003035.png" /> at <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003036.png" />,

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003037.png" /></td> </tr></table>

The influence function of an M-estimator is thus proportional to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003038.png" /> itself. Under suitable conditions, [[#References|[a3]]], M-estimators are asymptotically normal with asymptotic variance <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003039.png" />.

Optimal robust M-estimators can be obtained by solving Huber's minimax variance problem [[#References|[a1]]] or by minimizing the asymptotic variance <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003040.png" /> subject to an upper bound on the gross-error sensitivity <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003041.png" /> as in [[#References|[a2]]].

When estimating a univariate location, it is natural to use <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003042.png" />-functions of the type <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003043.png" />. The optimal robust M-estimator for univariate location at the Gaussian location model <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003044.png" /> (cf. also [[Gauss law|Gauss law]]) is given by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003045.png" />. This <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003046.png" /> has come to be known as Huber's function. Note that when <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003047.png" />, this M-estimator tends to the median (cf. also [[Median (in statistics)|Median (in statistics)]]), and when <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003048.png" /> it tends to the mean (cf. also [[Average|Average]]).

The breakdown value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003049.png" /> of an estimator <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003050.png" /> is the largest fraction of arbitrary outliers it can tolerate without becoming unbounded (see [[#References|[a2]]]). Any M-estimator with a monotone and bounded <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003051.png" /> function has breakdown value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003052.png" />, the highest possible value.

Location M-estimators are not invariant with respect to scale. Therefore it is recommended to compute <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003053.png" /> from

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003054.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a1)</td></tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003055.png" /> is a robust estimator of scale, e.g. the median absolute deviation

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003056.png" /></td> </tr></table>

which has <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003057.png" />.

For univariate scale estimation one uses <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003058.png" />-functions of the type <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003059.png" />. At the Gaussian scale model <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003060.png" />, the optimal robust M-estimators are given by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003061.png" />. For <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003062.png" /> one obtains the median absolute deviation and for <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003063.png" /> the standard deviation. In the general case, where both location and scale are unknown, one first computes <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003064.png" /> and then plugs it into (a1) for finding <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003065.png" />.

For multivariate location and scatter matrices, M-estimators were defined by R.A. Maronna [[#References|[a4]]], who also gave their influence function and asymptotic covariance matrix. For <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003066.png" />-dimensional data, the breakdown value of M-estimators is at most <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003067.png" />.

For [[Regression analysis|regression analysis]], one considers the linear model <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003068.png" /> where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003069.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003070.png" /> are column vectors, and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003071.png" /> and the error term <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003072.png" /> are independent. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003073.png" /> have a distribution with location zero and scale <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003074.png" />. For simplicity, put <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003075.png" />. Denote by <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003076.png" /> the joint distribution of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003077.png" />, which implies the distribution of the error term <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003078.png" />. Based on a data set <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003079.png" />, M-estimators <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003080.png" /> for regression [[#References|[a3]]] are defined by

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003081.png" /></td> </tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003082.png" /> are the residuals. If the Huber function <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003083.png" /> is used, the influence function of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003084.png" /> at <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003085.png" /> equals

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003086.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a2)</td></tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003087.png" />. The first factor of (a2) is the influence of the vertical error <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003088.png" />. It is bounded, which makes this estimator more robust than least squares (cf. also [[Least squares, method of|Least squares, method of]]). The second factor is the influence of the position <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003089.png" />. Unfortunately, this factor is unbounded, hence a single outlying <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003090.png" /> (i.e., a horizontal outlier) will almost completely determine the fit, as shown in [[#References|[a2]]]. Therefore the breakdown value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003091.png" />.

To obtain a bounded influence function, generalized M-estimators [[#References|[a2]]] are defined by

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003092.png" /></td> </tr></table>

for some real function <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003093.png" />. The influence function of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003094.png" /> at <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003095.png" /> now becomes

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003096.png" /></td> <td valign="top" style="width:5%;text-align:right;">(a3)</td></tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003097.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003098.png" />. For an appropriate choice of the function <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m12003099.png" />, the influence function (a3) is bounded, but still the breakdown value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030100.png" /> goes down to zero when the number of parameters <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030101.png" /> increases.

To repair this, P.J. Rousseeuw and V.J. Yohai [[#References|[a5]]] have introduced S-estimators. An S-estimator <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030102.png" /> minimizes <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030103.png" />, where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030104.png" /> are the residuals and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030105.png" /> is the robust scale estimator defined as the solution of

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030106.png" /></td> </tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030107.png" /> is taken to be <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030108.png" />. The function <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030109.png" /> must satisfy <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030110.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030111.png" /> and be continuously differentiable, and there must be a constant <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030112.png" /> such that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030113.png" /> is strictly increasing on <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030114.png" /> and constant on <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030115.png" />. Any S-estimator has breakdown value <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030116.png" /> in all dimensions, and it is asymptotically normal with the same asymptotic covariance as the M-estimator with that function <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/m/m120/m120030/m120030117.png" />. The S-estimators have also been generalized to multivariate location and scatter matrices, in [[#References|[a6]]], and they enjoy the same properties.

====References====
<table><TR><TD valign="top">[a1]</TD> <TD valign="top">  P.J. Huber,   "Robust estimation of a location parameter"  ''Ann. Math. Stat.'' , '''35'''  (1964)  pp. 73–101</TD></TR><TR><TD valign="top">[a2]</TD> <TD valign="top">  F.R. Hampel,   E.M. Ronchetti,   P.J. Rousseeuw,   W.A. Stahel,   "Robust statistics: The approach based on influence functions" , Wiley  (1986)</TD></TR><TR><TD valign="top">[a3]</TD> <TD valign="top">  P.J. Huber,   "Robust statistics" , Wiley  (1981)</TD></TR><TR><TD valign="top">[a4]</TD> <TD valign="top">  R.A. Maronna,   "Robust M-estimators of multivariate location and scatter"  ''Ann. Statist.'' , '''4'''  (1976)  pp. 51–67</TD></TR><TR><TD valign="top">[a5]</TD> <TD valign="top">  P.J. Rousseeuw,   V.J. Yohai,   "Robust regression by means of S-estimators"  J. Franke (ed.)  W. Härdle (ed.)  R.D. Martin (ed.) , ''Robust and Nonlinear Time Ser. Analysis'' , ''Lecture Notes Statistics'' , '''26''' , Springer  (1984)  pp. 256–272</TD></TR><TR><TD valign="top">[a6]</TD> <TD valign="top">  P.J. Rousseeuw,   A. Leroy,   "Robust regression and outlier detection" , Wiley  (1987)</TD></TR></table>