Information correlation coefficient

The measure of dependency between two random variables and defined as a function of the amount of information in one random variable with respect to the other by:

where is the amount of information (cf. Information, amount of).

The properties of as a measure of dependency are completely determined by the properties of , which is itself a characteristic of the dependence between and . However, the use of as a measure of dependency and as the information analogue of the correlation coefficient is justified by the fact that for arbitrary random variables it has the advantage over that, because of the properties of information, if and only if and are independent. If and have a joint normal distribution, then these two coefficients coincide, since in this case

The practical investigation of dependence by the information correlation coefficient is equivalent to the analysis of the amount of information in tables of the type of contingency tables of tests. The sample analogue of is the coefficient

computed in terms of the information statistic :

where is the number of observations, and are the numbers of grouping classes by the two characteristics, is the number of observations in the class , , . Thus, the problem of the distribution of the sample information coefficient leads to the problem of the distribution of the sample information. The analysis of the sample information as a measure of dependency is made difficult by the fact that strongly depends on the grouping of the observations.

References

[1]	E. Linfoot, "An informational measure of correlation" Information and Control , 1 : 1 (1957) pp. 85–89
[2]	S. Kullback, "Information theory and statistics" , Wiley (1959)