View source for Differential entropy

<!--
d0318901.png
$#A+1 = 34 n = 0
$#C+1 = 34 : ~/encyclopedia/old_files/data/D031/D.0301890 Differential entropy
Automatically converted into TeX, above some diagnostics.
Please remove this comment and the {{TEX|auto}} line below,
if TeX found to be correct.
-->

{{TEX|auto}}
{{TEX|done}}

The formal analogue of the concept of entropy for random variables having distribution densities. The differential entropy  $  h ( \xi ) $
of a random variable  $  \xi $
defined on some probability space  $  ( \Omega , \mathfrak A , P) $, 
assuming values in an  $  n $-
dimensional Euclidean space  $  \mathbf R  ^ {n} $
and having distribution density  $  p( x) $, 
$  x \in \mathbf R  ^ {n} $, 
is given by the formula

$$ 
h ( \xi )  =  \int\limits _ {\mathbf R  ^ {n} } p ( x)   \mathop{\rm log}  p ( x)  dx ,
$$

where  $  0   \mathop{\rm log}  0 $
is assumed to be equal to zero. Thus, the differential entropy coincides with the entropy of the measure  $  P ( \cdot ) $
with respect to the Lebesgue measure  $  \lambda ( \cdot ) $, 
where  $  P ( \cdot ) $
is the distribution of  $  \xi $.

The concept of the differential entropy proves useful in computing various information-theoretic characteristics, in the first place the mutual amount of information (cf. [[Information, amount of|Information, amount of]])  $  J ( \xi , \eta ) $
of two random vectors  $  \xi $
and  $  \eta $. 
If  $  h ( \xi ) $, 
$  h ( \eta ) $
and  $  h ( \xi , \eta ) $(
i.e. the differential entropy of the pair  $  ( \xi , \eta ) $) 
are finite, the following formula is valid:

$$ 
J ( \xi , \eta )  =  - h ( \xi , \eta ) + h ( \xi )+ h ( \eta ).
$$

The following two properties of the differential entropy are worthy of mention: 1) as distinct from the ordinary entropy, the differential entropy is not covariant with respect to a change in the coordinate system and may assume negative values; and 2) let  $  \phi ( \xi ) $
be the discretization of an  $  n $-
dimensional random variable  $  \xi $
having a density, with steps of  $  \Delta x $; 
then for the entropy  $  H ( \phi ( x)) $
the formula

$$ 
H ( \phi ( \xi ))  =  - n   \mathop{\rm log}  \Delta x + h ( \xi ) + o ( 1)
$$

is valid as  $  \Delta \rightarrow 0 $. 
Thus,  $  H ( \phi ( x )) \rightarrow + \infty $
as  $  \Delta x \rightarrow 0 $. 
The principal term of the asymptotics of  $  H ( \phi ( \xi )) $
depends on the dimension of the space of values of  $  \xi $. 
The differential entropy defines the term next in order of the asymptotic expansion independent of  $  \Delta x $
and it is the first term involving a dependence on the actual nature of the distribution of  $  \xi $.

====References====
<table><TR><TD valign="top">[1]</TD> <TD valign="top">  I.M. Gel'fand,   A.N. Kolmogorov,   A.M. Yaglom,   "The amount of information in, and entropy of, continuous distributions" , ''Proc. 3-rd All-Union Math. Congress'' , '''3''' , Moscow  (1958)  pp. 300–320  (In Russian)</TD></TR><TR><TD valign="top">[2]</TD> <TD valign="top">  A. Rényi,   "Wahrscheinlichkeitsrechnung" , Deutsch. Verlag Wissenschaft.  (1962)</TD></TR></table>