Differential entropy
The formal analogue of the concept of entropy for random variables having distribution densities. The differential entropy of a random variable
defined on some probability space
, assuming values in an
-dimensional Euclidean space
and having distribution density
,
, is given by the formula
![]() |
where is assumed to be equal to zero. Thus, the differential entropy coincides with the entropy of the measure
with respect to the Lebesgue measure
, where
is the distribution of
.
The concept of the differential entropy proves useful in computing various information-theoretic characteristics, in the first place the mutual amount of information (cf. Information, amount of) of two random vectors
and
. If
,
and
(i.e. the differential entropy of the pair
) are finite, the following formula is valid:
![]() |
The following two properties of the differential entropy are worthy of mention: 1) as distinct from the ordinary entropy, the differential entropy is not covariant with respect to a change in the coordinate system and may assume negative values; and 2) let be the discretization of an
-dimensional random variable
having a density, with steps of
; then for the entropy
the formula
![]() |
is valid as . Thus,
as
. The principal term of the asymptotics of
depends on the dimension of the space of values of
. The differential entropy defines the term next in order of the asymptotic expansion independent of
and it is the first term involving a dependence on the actual nature of the distribution of
.
References
[1] | I.M. Gel'fand, A.N. Kolmogorov, A.M. Yaglom, "The amount of information in, and entropy of, continuous distributions" , Proc. 3-rd All-Union Math. Congress , 3 , Moscow (1958) pp. 300–320 (In Russian) |
[2] | A. Rényi, "Wahrscheinlichkeitsrechnung" , Deutsch. Verlag Wissenschaft. (1962) |
Differential entropy. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Differential_entropy&oldid=46665