Correlation (in statistics)

A dependence between random variables not necessarily expressed by a rigorous functional relationship. Unlike functional dependence, a correlation is, as a rule, considered when one of the random variables depends not only on the other (given) one, but also on several random factors. The dependence between two random events is manifested in the fact that the conditional probability of one of them, given the occurrence of the other, differs from the unconditional probability. Similarly, the influence of one random variable on another is characterized by the conditional distributions of one of them, given fixed values of the other. Let $ X $ and $ Y $ be random variables with given joint distribution, let $ m _ {X} $ and $ m _ {Y} $ be the expectations of $ X $ and $ Y $, let $ \sigma _ {X} ^ {2} $ and $ \sigma _ {Y} ^ {2} $ be the variances of $ X $ and $ Y $, and let $ \rho $ be the correlation coefficient of $ X $ and $ Y $. Assume that for every possible value $ X = x $ the conditional mathematical expectation $ y ( x) = {\mathsf E} [ Y \mid X = x] $ of $ Y $ is defined; then the function $ y ( x) $ is known as the regression of $ Y $ given $ X $, and its graph is the regression curve of $ Y $ given $ X $. The dependence of $ Y $ on $ X $ is manifested in the variation of the mean values of $ Y $ as $ X $ varies, although for each fixed value $ X = x $, $ Y $ remains a random variable with a well-defined spread. In order to determine to what degree of accuracy the regression reproduces the variation of $ Y $ as $ X $ varies, one uses the conditional variance of $ Y $ for a given $ X = x $ or its mean value (a measure of the spread of $ Y $ about the regression curve):

$$ \sigma _ {Y \mid X } ^ {2} = \ {\mathsf E} [ Y - {\mathsf E} ( Y \mid X = x)] ^ {2} . $$

If $ X $ and $ Y $ are independent, then all conditional mathematical expectations of $ Y $ are independent of $ x $ and coincide with the unconditional expectations: $ y ( x) = m _ {Y} $; and then also $ \sigma _ {Y \mid X } ^ {2} = \sigma _ {Y} ^ {2} $. When $ Y $ is a function of $ X $ in the strict sense of the word, then for each $ X = x $ the variable $ Y $ takes only one definite value and $ \sigma _ {Y \mid X } ^ {2} = 0 $. Similarly one defines $ x ( y) = {\mathsf E} [ X \mid Y = y] $( the regression of $ X $ given $ Y $). A natural index of the concentration of the distribution near the regression curve $ y ( x) $ is the correlation ratio

$$ \eta _ {Y \mid X } ^ {2} = \ 1 - \frac{\sigma _ {Y \mid X } ^ {2} }{\sigma _ {Y} ^ {2} } . $$

One has $ \eta _ {Y \mid X } ^ {2} = 0 $ if and only if the regression has the form $ y ( x) = m _ {Y} $, and in that case the correlation coefficient $ \rho $ vanishes and $ Y $ is not correlated with $ X $. If the regression of $ Y $ given $ X $ is linear, i.e. the regression curve is the straight line

$$ y ( x) = m _ {y} + \rho \frac{\sigma _ {Y} }{\sigma _ {X} } ( x - m _ {X} ), $$

then

$$ \sigma _ {Y \mid X } ^ {2} = \ \sigma _ {Y} ^ {2} ( 1 - \rho ^ {2} ) \ \ \textrm{ and } \ \ \eta _ {Y \mid X } ^ {2} = \rho ^ {2} ; $$

if, moreover, $ | \rho | = 1 $, then $ Y $ is related to $ X $ through an exact linear dependence; but if $ \eta _ {Y \mid X } ^ {2} = \rho ^ {2} < 1 $, there is no functional dependence between $ Y $ and $ X $. There is an exact functional dependence of $ Y $ on $ X $, other than a linear one, if and only if $ \rho ^ {2} < \eta _ {Y \mid X } ^ {2} = 1 $. With rare exceptions, the practical use of the correlation coefficient as a measure of the lack of dependence is justifiable only when the joint distribution of $ X $ and $ Y $ is normal (or close to normal), since in that case $ \rho = 0 $ implies that $ X $ and $ Y $ are independent. Use of $ \rho $ as a measure of dependence for arbitrary random variables $ X $ and $ Y $ frequently leads to erroneous conclusions, since $ \rho $ may vanish even when a functional dependence exists. If the joint distribution of $ X $ and $ Y $ is normal, then both regression curves are straight lines and $ \rho $ uniquely determines the concentration of the distribution near the regression curves: When $ | \rho | = 1 $ the regression curves merge into one, corresponding to linear dependence between $ X $ and $ Y $; when $ \rho = 0 $ one has independence.

When studying the interdependence of several random variables $ X _ {1} \dots X _ {n} $ with a given joint distribution, one uses multiple and partial correlation ratios and coefficients. The latter are evaluated using the ordinary correlation coefficients between $ X _ {i} $ and $ X _ {j} $, the totality of which form the correlation matrix. A measure of the linear relationship between $ X _ {1} $ and the totality of the other variables $ X _ {2} \dots X _ {n} $ is provided by the multiple-correlation coefficient. If the mutual relationship of $ X _ {1} $ and $ X _ {2} $ is assumed to be determined by the influence of the other variables $ X _ {3} \dots X _ {n} $, then the partial correlation coefficient of $ X _ {1} $ and $ X _ {2} $ with respect to $ X _ {3} \dots X _ {n} $ is an index of the linear relationship between $ X _ {1} $ and $ X _ {2} $ relative to $ X _ {3} \dots X _ {n} $.

For measures of correlation based on rank statistics (cf. Rank statistic) see Kendall coefficient of rank correlation; Spearman coefficient of rank correlation.

Mathematical statisticians have developed methods for estimating coefficients that characterize the correlation between random variables or tests; there are also methods to test hypotheses concerning their values, using their sampling analogues. These methods are collectively known as correlation analysis. Correlation analysis of statistical data consists of the following basic practical steps: 1) the construction of a scatter plot and the compilation of a correlation table; 2) the computation of sampling correlation ratios or correlation coefficients; 3) testing statistical hypothesis concerning the significance of the dependence. Further investigation may consist in establishing the concrete form of the dependence between the variables (see Regression).

Among the aids to analysis of two-dimensional sample data are the scatter plot and the correlation table. The scatter plot is obtained by plotting the sample points on the coordinate plane. Examination of the configuration formed by the points of the scatter plot yields a preliminary idea of the type of dependence between the random variables (e.g. whether one of the variables increases or decreases on the average as the other increases). Prior to numerical processing, the results are usually grouped and presented in the form of a correlation table. In each entry of this table one writes the number $ n _ {ij} $ of pairs $ ( x, y) $ with components in the appropriate grouping intervals. Assuming that the grouping intervals (in each of the variables) are equal in length, one takes the centres $ x _ {i} $( or $ y _ {i} $) of the intervals and the numbers $ n _ {ij} $ as the basis for calculation.

For more accurate information about the nature and strength of the relationship than that provided by the scatter plot, one turns to the correlation coefficient and correlation ratio. The sample correlation coefficient is defined by the formula

$$ \widehat \rho = \ \frac{\sum _ { i } \sum _ { j } ( x _ {i} - \overline{x}\; ) ( y _ {j} - \overline{y}\; ) n _ {ij} }{\sqrt {\sum _ { i } n _ {i \cdot } ( x _ {i} - \overline{x}\; ) ^ {2} } \sqrt {\sum _ { j } n _ {\cdot j } ( y _ {j} - \overline{y}\; ) ^ {2} } } , $$

where

$$ n _ {i \cdot } = \ \sum _ { j } n _ {ij} ,\ \ n _ {\cdot j } = \ \sum _ { i } n _ {ij} $$

and

$$ \overline{x}\; = \ \frac{\sum _ { i } n _ {i \cdot } x _ {i} }{n} ,\ \ \overline{y}\; = \ \frac{\sum _ { j } n _ {\cdot j } y _ {j} }{n} . $$

In the case of a large number of independent observations, governed by one and the same near-normal distribution, $ \widehat \rho $ is a good approximation to the true correlation coefficient $ \rho $. In all other cases, as characteristic of strength of the relationship the correlation ratio is recommended, the interpretation of which is independent of the type of dependence being studied. The sample value $ \widehat \eta {} _ {Y \mid X } ^ {2} $ is computed from the entries in the correlation table:

$$ \widehat \eta {} _ {Y \mid X } ^ {2} = \ \frac{ { \frac{1}{n} } \sum _ { i } n _ {i \cdot } ( \overline{y}\; _ {i} - \overline{y}\; ) ^ {2} }{ { \frac{1}{n} } \sum _ { j } n _ {\cdot j } ( y _ {j} - \overline{y}\; ) ^ {2} } , $$

where the numerator represents the spread of the conditional mean values $ \overline{y}\; _ {i} $ about the unconditional mean $ \overline{y}\; $( the sample value $ \widehat \eta {} _ {X \mid Y } ^ {2} $ is defined analogously). The quantity $ \widehat \eta {} _ {Y \mid X } ^ {2} - \widehat \rho {} ^ {2} $ is used as an indicator of the deviation of the regression from linearity.

The testing of hypotheses concerning the significance of a relationship are based on the distributions of the sample correlation characteristics. In the case of a normal distribution, the value of the sample correlation coefficient $ \widehat \rho $ is significantly distinct from zero if

$$ ( \widehat \rho ) ^ {2} > \ \left [ 1 + \frac{n - 2 }{t _ \alpha ^ {2} } \right ] ^ {-} 1 , $$

where $ t _ \alpha $ is the critical value of the Student $ t $- distribution with $ ( n - 2) $ degrees of freedom corresponding to the chosen significance level $ \alpha $. If $ \rho \neq 0 $ one usually uses the Fisher $ z $- transform, with $ \widehat \rho $ replaced by $ z $ according to the formula

$$ z = { \frac{1}{2} } \mathop{\rm ln} \left ( \frac{1 + \widehat \rho }{1 - \widehat \rho } \right ) . $$

Even at relatively small values $ n $ the distribution of $ z $ is a good approximation to the normal distribution with mathematical expectation

$$ \frac{1}{2} \mathop{\rm ln} \frac{1+ \rho }{1 - \rho } + \frac \rho {2( n - 1) } $$

and variance $ 1/( n - 3) $. On this basis one can now define approximate confidence intervals for the true correlation coefficient $ \rho $.

For the distribution of the sample correlation ratio and for tests of the linearity hypothesis for the regression, see [3].

References

[1]	H. Cramér, "Mathematical methods of statistics" , Princeton Univ. Press (1946)
[2]	B.L. van der Waerden, "Mathematische Statistik" , Springer (1957)
[3]	M.G. Kendall, A. Stuart, "The advanced theory of statistics" , 2. Inference and relationship , Griffin (1979)
[4]	S.A. Aivazyan, "Statistical research on dependence" , Moscow (1968) (In Russian)