Differential entropy

The formal analogue of the concept of entropy for random variables having distribution densities. The differential entropy $ h ( \xi ) $ of a random variable $ \xi $ defined on some probability space $ ( \Omega , \mathfrak A , P) $, assuming values in an $ n $- dimensional Euclidean space $ \mathbf R ^ {n} $ and having distribution density $ p( x) $, $ x \in \mathbf R ^ {n} $, is given by the formula

$$ h ( \xi ) = \int\limits _ {\mathbf R ^ {n} } p ( x) \mathop{\rm log} p ( x) dx , $$

where $ 0 \mathop{\rm log} 0 $ is assumed to be equal to zero. Thus, the differential entropy coincides with the entropy of the measure $ P ( \cdot ) $ with respect to the Lebesgue measure $ \lambda ( \cdot ) $, where $ P ( \cdot ) $ is the distribution of $ \xi $.

The concept of the differential entropy proves useful in computing various information-theoretic characteristics, in the first place the mutual amount of information (cf. Information, amount of) $ J ( \xi , \eta ) $ of two random vectors $ \xi $ and $ \eta $. If $ h ( \xi ) $, $ h ( \eta ) $ and $ h ( \xi , \eta ) $( i.e. the differential entropy of the pair $ ( \xi , \eta ) $) are finite, the following formula is valid:

$$ J ( \xi , \eta ) = - h ( \xi , \eta ) + h ( \xi )+ h ( \eta ). $$

The following two properties of the differential entropy are worthy of mention: 1) as distinct from the ordinary entropy, the differential entropy is not covariant with respect to a change in the coordinate system and may assume negative values; and 2) let $ \phi ( \xi ) $ be the discretization of an $ n $- dimensional random variable $ \xi $ having a density, with steps of $ \Delta x $; then for the entropy $ H ( \phi ( x)) $ the formula

$$ H ( \phi ( \xi )) = - n \mathop{\rm log} \Delta x + h ( \xi ) + o ( 1) $$

is valid as $ \Delta \rightarrow 0 $. Thus, $ H ( \phi ( x )) \rightarrow + \infty $ as $ \Delta x \rightarrow 0 $. The principal term of the asymptotics of $ H ( \phi ( \xi )) $ depends on the dimension of the space of values of $ \xi $. The differential entropy defines the term next in order of the asymptotic expansion independent of $ \Delta x $ and it is the first term involving a dependence on the actual nature of the distribution of $ \xi $.

References

[1]	I.M. Gel'fand, A.N. Kolmogorov, A.M. Yaglom, "The amount of information in, and entropy of, continuous distributions" , Proc. 3-rd All-Union Math. Congress , 3 , Moscow (1958) pp. 300–320 (In Russian)
[2]	A. Rényi, "Wahrscheinlichkeitsrechnung" , Deutsch. Verlag Wissenschaft. (1962)