Namespaces
Variants
Actions

Difference between revisions of "Information correlation coefficient"

From Encyclopedia of Mathematics
Jump to: navigation, search
(Importing text file)
 
m (tex encoded by computer)
Line 1: Line 1:
The measure of dependency between two random variables <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510601.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510602.png" /> defined as a function of the amount of information in one random variable with respect to the other by:
+
<!--
 +
i0510601.png
 +
$#A+1 = 29 n = 0
 +
$#C+1 = 29 : ~/encyclopedia/old_files/data/I051/I.0501060 Information correlation coefficient
 +
Automatically converted into TeX, above some diagnostics.
 +
Please remove this comment and the {{TEX|auto}} line below,
 +
if TeX found to be correct.
 +
-->
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510603.png" /></td> </tr></table>
+
{{TEX|auto}}
 +
{{TEX|done}}
  
where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510604.png" /> is the amount of information (cf. [[Information, amount of|Information, amount of]]).
+
The measure of dependency between two random variables  $  X $
 +
and  $  Y $
 +
defined as a function of the amount of information in one random variable with respect to the other by:
  
The properties of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510605.png" /> as a measure of dependency are completely determined by the properties of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510606.png" />, which is itself a characteristic of the dependence between <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510607.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510608.png" />. However, the use of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i0510609.png" /> as a measure of dependency and as the information analogue of the [[Correlation coefficient|correlation coefficient]] <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106010.png" /> is justified by the fact that for arbitrary random variables it has the advantage over <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106011.png" /> that, because of the properties of information, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106012.png" /> if and only if <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106013.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106014.png" /> are independent. If <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106015.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106016.png" /> have a joint normal distribution, then these two coefficients coincide, since in this case
+
$$
 +
R ( X , Y )  = \sqrt {1 - e ^ {- 2I( X, Y) } } ,
 +
$$
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106017.png" /></td> </tr></table>
+
where  $  I ( X , Y ) $
 +
is the amount of information (cf. [[Information, amount of|Information, amount of]]).
  
The practical investigation of dependence by the information correlation coefficient is equivalent to the analysis of the amount of information in tables of the type of contingency tables of tests. The sample analogue of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106018.png" /> is the coefficient
+
The properties of $  R ( X , Y ) $
 +
as a measure of dependency are completely determined by the properties of  $  I ( X , Y ) $,
 +
which is itself a characteristic of the dependence between  $  X $
 +
and  $  Y $.
 +
However, the use of  $  R ( X , Y ) $
 +
as a measure of dependency and as the information analogue of the [[Correlation coefficient|correlation coefficient]]  $  \rho $
 +
is justified by the fact that for arbitrary random variables it has the advantage over  $  \rho $
 +
that, because of the properties of information,  $  R ( X , Y ) = 0 $
 +
if and only if  $  X $
 +
and  $  Y $
 +
are independent. If  $  X $
 +
and  $  Y $
 +
have a joint normal distribution, then these two coefficients coincide, since in this case
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106019.png" /></td> </tr></table>
+
$$
 +
I( X, Y)  = -
 +
\frac{1}{2}
 +
  \mathop{\rm ln}  ( 1 - \rho  ^ {2} ) .
 +
$$
  
computed in terms of the information statistic <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106020.png" />:
+
The practical investigation of dependence by the information correlation coefficient is equivalent to the analysis of the amount of information in tables of the type of contingency tables of tests. The sample analogue of  $  R $
 +
is the coefficient
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106021.png" /></td> </tr></table>
+
$$
 +
\widehat{R}  = \sqrt {1 - e ^ {- 2 \widehat{I}  } } ,
 +
$$
  
where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106022.png" /> is the number of observations, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106023.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106024.png" /> are the numbers of grouping classes by the two characteristics, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106025.png" /> is the number of observations in the class <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106026.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106027.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106028.png" />. Thus, the problem of the distribution of the sample information coefficient leads to the problem of the distribution of the sample information. The analysis of the sample information as a measure of dependency is made difficult by the fact that <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051060/i05106029.png" /> strongly depends on the grouping of the observations.
+
computed in terms of the information statistic  $  \widehat{I}  $:
 +
 
 +
$$
 +
\widehat{I}  = \
 +
\sum _ { i= } 1 ^ { s }  \sum _ { j= } 1 ^ { t }
 +
 
 +
\frac{n _ {ij} }{n}
 +
\
 +
\mathop{\rm ln} \
 +
 
 +
\frac{n n _ {ij} }{n _ {i \cdot }  n _ {\cdot j }  }
 +
,
 +
$$
 +
 
 +
where  $  n $
 +
is the number of observations, $  s $
 +
and $  t $
 +
are the numbers of grouping classes by the two characteristics, $  n _ {ij} $
 +
is the number of observations in the class $  ( i , j ) $,
 +
$  n _ {i \cdot }  = \sum _ {j=} 1  ^ {t} n _ {ij} $,
 +
$  n _ {\cdot j }  = \sum _ {i=} 1  ^ {s} n _ {ij} $.  
 +
Thus, the problem of the distribution of the sample information coefficient leads to the problem of the distribution of the sample information. The analysis of the sample information as a measure of dependency is made difficult by the fact that $  \widehat{I}  $
 +
strongly depends on the grouping of the observations.
  
 
====References====
 
====References====
 
<table><TR><TD valign="top">[1]</TD> <TD valign="top">  E. Linfoot,  "An informational measure of correlation"  ''Information and Control'' , '''1''' :  1  (1957)  pp. 85–89</TD></TR><TR><TD valign="top">[2]</TD> <TD valign="top">  S. Kullback,  "Information theory and statistics" , Wiley  (1959)</TD></TR></table>
 
<table><TR><TD valign="top">[1]</TD> <TD valign="top">  E. Linfoot,  "An informational measure of correlation"  ''Information and Control'' , '''1''' :  1  (1957)  pp. 85–89</TD></TR><TR><TD valign="top">[2]</TD> <TD valign="top">  S. Kullback,  "Information theory and statistics" , Wiley  (1959)</TD></TR></table>

Revision as of 22:12, 5 June 2020


The measure of dependency between two random variables $ X $ and $ Y $ defined as a function of the amount of information in one random variable with respect to the other by:

$$ R ( X , Y ) = \sqrt {1 - e ^ {- 2I( X, Y) } } , $$

where $ I ( X , Y ) $ is the amount of information (cf. Information, amount of).

The properties of $ R ( X , Y ) $ as a measure of dependency are completely determined by the properties of $ I ( X , Y ) $, which is itself a characteristic of the dependence between $ X $ and $ Y $. However, the use of $ R ( X , Y ) $ as a measure of dependency and as the information analogue of the correlation coefficient $ \rho $ is justified by the fact that for arbitrary random variables it has the advantage over $ \rho $ that, because of the properties of information, $ R ( X , Y ) = 0 $ if and only if $ X $ and $ Y $ are independent. If $ X $ and $ Y $ have a joint normal distribution, then these two coefficients coincide, since in this case

$$ I( X, Y) = - \frac{1}{2} \mathop{\rm ln} ( 1 - \rho ^ {2} ) . $$

The practical investigation of dependence by the information correlation coefficient is equivalent to the analysis of the amount of information in tables of the type of contingency tables of tests. The sample analogue of $ R $ is the coefficient

$$ \widehat{R} = \sqrt {1 - e ^ {- 2 \widehat{I} } } , $$

computed in terms of the information statistic $ \widehat{I} $:

$$ \widehat{I} = \ \sum _ { i= } 1 ^ { s } \sum _ { j= } 1 ^ { t } \frac{n _ {ij} }{n} \ \mathop{\rm ln} \ \frac{n n _ {ij} }{n _ {i \cdot } n _ {\cdot j } } , $$

where $ n $ is the number of observations, $ s $ and $ t $ are the numbers of grouping classes by the two characteristics, $ n _ {ij} $ is the number of observations in the class $ ( i , j ) $, $ n _ {i \cdot } = \sum _ {j=} 1 ^ {t} n _ {ij} $, $ n _ {\cdot j } = \sum _ {i=} 1 ^ {s} n _ {ij} $. Thus, the problem of the distribution of the sample information coefficient leads to the problem of the distribution of the sample information. The analysis of the sample information as a measure of dependency is made difficult by the fact that $ \widehat{I} $ strongly depends on the grouping of the observations.

References

[1] E. Linfoot, "An informational measure of correlation" Information and Control , 1 : 1 (1957) pp. 85–89
[2] S. Kullback, "Information theory and statistics" , Wiley (1959)
How to Cite This Entry:
Information correlation coefficient. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Information_correlation_coefficient&oldid=47355
This article was adapted from an original article by A.V. Prokhorov (originator), which appeared in Encyclopedia of Mathematics - ISBN 1402006098. See original article