View source for Information, amount of

<!--
i0510501.png
$#A+1 = 45 n = 0
$#C+1 = 45 : ~/encyclopedia/old_files/data/I051/I.0501050 Information, amount of
Automatically converted into TeX, above some diagnostics.
Please remove this comment and the {{TEX|auto}} line below,
if TeX found to be correct.
-->

{{TEX|auto}}
{{TEX|done}}

An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let  $  \xi $
and  $  \eta $
be random variables defined on a [[Probability space|probability space]]  $  ( \Omega , \mathfrak A , {\mathsf P} ) $
and taking values in measurable spaces (cf. [[Measurable space|Measurable space]])  $  ( \mathfrak X , S _ {\mathfrak X }  ) $
and  $  ( \mathfrak Y , S _ {\mathfrak Y }  ) $, 
respectively. Let  $  p _ {\xi \eta }  ( C) $, 
$  C \in S _ {\mathfrak X }  \times S _ {\mathfrak Y }  $, 
and  $  p _  \xi  ( A) $, 
$  A \in S _ {\mathfrak X }  $, 
$  p _  \eta  ( B) $, 
$  B \in S _ {\mathfrak Y }  $, 
be their joint and marginale probability distributions. If  $  p _ {\xi \eta }  ( \cdot ) $
is absolutely continuous with respect to the direct product of measures  $  p _  \xi  \times p _  \eta  ( \cdot ) $, 
if  $  a _ {\xi \eta }  ( \cdot ) $
is the (Radon–Nikodým) density of  $  p _ {\xi \eta }  ( \cdot ) $
with respect to  $  p _  \xi  \times p _  \eta  ( \cdot ) $, 
and if  $  i _ {\xi \eta }  ( \cdot ) =  \mathop{\rm log}  a _ {\xi \eta }  ( \cdot ) $
is the information density (the logarithms are usually taken to base 2 or  $  e $), 
then, by definition, the amount of information is given by

$$ 
I ( \xi , \eta )  = \ 
\int\limits _ {\mathfrak X \times \mathfrak Y }
i _ {\xi \eta }  ( x , y )  p _ {\xi \eta }  ( d x , d y ) =
$$

$$ 
= \ 
\int\limits _ {\mathfrak X \times \mathfrak Y } a _ {\xi \eta }  ( x , y )   \mathop{\rm log} \ 
a _ {\xi \eta }  ( x , y )  p _  \xi  ( d x )  p _  \eta  ( d y ) .
$$

If  $  p _ {\xi \eta }  ( \cdot ) $
is not absolutely continuous with respect to  $  p _  \xi  \times p _  \eta  ( \cdot ) $, 
then  $  I ( \xi , \eta ) = + \infty $, 
by definition.

In case the random variables  $  \xi $
and  $  \eta $
take only a finite number of values, the expression for  $  I ( \xi , \eta ) $
takes the form

$$ 
I ( \xi , \eta )  = \ 
\sum _ { i= } 1 ^ { n }  \sum _ { j= } 1 ^ { m } 
p _ {ij}   \mathop{\rm log} \ 

\frac{p _ {ij} }{p _ {i} q _ {i} }
 ,
$$

where

$$ 
\{ p _ {i} \} _ {i=} 1  ^ {n} ,\ \ 
\{ q _ {j} \} _ {j=} 1  ^ {m} ,\ \ 
\{ {p _ {ij} } : {i = 1 \dots n ;  j = 1 \dots m } \}
$$

are the probability functions of  $  \xi $, 
$  \eta $
and the pair  $  ( \xi , \eta ) $, 
respectively. (In particular,

$$ 
I ( \xi , \xi )  =  -
\sum _ { i= } 1 ^ { n } 
p _ {i}   \mathop{\rm log}  p _ {i}  =  H ( \xi )
$$

is the [[Entropy|entropy]] of  $  \xi $.) 
In case  $  \xi $
and  $  \eta $
are random vectors and the densities  $  p _  \xi  ( x) $, 
$  p _  \eta  ( y) $
and  $  p _ {\xi \eta }  ( x , y ) $
of  $  \xi $, 
$  \eta $
and the pair  $  ( \xi , \eta ) $, 
respectively, exist, one has

$$ 
I ( \xi , \eta )  = \ 
\int\limits p _ {\xi \eta }  ( x , y )   \mathop{\rm log}  
\frac{p _ {\xi \eta } 
( x , y ) }{p _  \xi  ( x) p _  \eta  ( y) }
 \ 
d x  d y .
$$

In general,

$$ 
I ( \xi , \eta )  = \ 
\sup  I ( \phi ( \xi ) , \psi ( \eta ) ) ,
$$

where the supremum is over all measurable functions  $  \phi ( \cdot ) $
and  $  \psi ( \cdot ) $
with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.

For references, see , ,

to [[Information, transmission of|Information, transmission of]].