Namespaces
Variants
Actions

Difference between revisions of "Information, amount of"

From Encyclopedia of Mathematics
Jump to: navigation, search
(Importing text file)
 
(latex details)
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510501.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510502.png" /> be random variables defined on a [[Probability space|probability space]] <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510503.png" /> and taking values in measurable spaces (cf. [[Measurable space|Measurable space]]) <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510504.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510505.png" />, respectively. Let <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510506.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510507.png" />, and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510508.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i0510509.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105010.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105011.png" />, be their joint and marginale probability distributions. If <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105012.png" /> is absolutely continuous with respect to the direct product of measures <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105013.png" />, if <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105014.png" /> is the (Radon–Nikodým) density of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105015.png" /> with respect to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105016.png" />, and if <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105017.png" /> is the information density (the logarithms are usually taken to base 2 or <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105018.png" />), then, by definition, the amount of information is given by
+
<!--
 +
i0510501.png
 +
$#A+1 = 45 n = 0
 +
$#C+1 = 45 : ~/encyclopedia/old_files/data/I051/I.0501050 Information, amount of
 +
Automatically converted into TeX, above some diagnostics.
 +
Please remove this comment and the {{TEX|auto}} line below,
 +
if TeX found to be correct.
 +
-->
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105019.png" /></td> </tr></table>
+
{{TEX|auto}}
 +
{{TEX|done}}
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105020.png" /></td> </tr></table>
+
An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let  $  \xi $
 +
and  $  \eta $
 +
be random variables defined on a [[Probability space|probability space]]  $  ( \Omega , \mathfrak A , {\mathsf P} ) $
 +
and taking values in measurable spaces (cf. [[Measurable space|Measurable space]])  $  ( \mathfrak X , S _ {\mathfrak X }  ) $
 +
and  $  ( \mathfrak Y , S _ {\mathfrak Y }  ) $,
 +
respectively. Let  $  p _ {\xi \eta }  ( C) $,
 +
$  C \in S _ {\mathfrak X }  \times S _ {\mathfrak Y }  $,
 +
and  $  p _  \xi  ( A) $,
 +
$  A \in S _ {\mathfrak X }  $,
 +
$  p _  \eta  ( B) $,
 +
$  B \in S _ {\mathfrak Y }  $,
 +
be their joint and marginale probability distributions. If  $  p _ {\xi \eta }  ( \cdot ) $
 +
is absolutely continuous with respect to the direct product of measures  $  p _  \xi  \times p _  \eta  ( \cdot ) $,
 +
if  $  a _ {\xi \eta }  ( \cdot ) $
 +
is the (Radon–Nikodým) density of  $  p _ {\xi \eta }  ( \cdot ) $
 +
with respect to  $  p _  \xi  \times p _  \eta  ( \cdot ) $,
 +
and if  $  i _ {\xi \eta }  ( \cdot ) =  \mathop{\rm log}  a _ {\xi \eta }  ( \cdot ) $
 +
is the information density (the logarithms are usually taken to base 2 or  $  e $),
 +
then, by definition, the amount of information is given by
  
If <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105021.png" /> is not absolutely continuous with respect to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105022.png" />, then <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105023.png" />, by definition.
+
$$
 +
I ( \xi , \eta )  = \
 +
\int\limits _ {\mathfrak X \times \mathfrak Y }
 +
i _ {\xi \eta }  ( x , y )  p _ {\xi \eta }  ( d x , d y ) =
 +
$$
  
In case the random variables <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105024.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105025.png" /> take only a finite number of values, the expression for <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105026.png" /> takes the form
+
$$
 +
= \
 +
\int\limits _ {\mathfrak X \times \mathfrak Y } a _ {\xi \eta }  ( x , y )  \mathop{\rm log} \
 +
a _ {\xi \eta }  ( x , y )  p _  \xi  ( d x )  p _  \eta  ( d y ) .
 +
$$
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105027.png" /></td> </tr></table>
+
If  $  p _ {\xi \eta }  ( \cdot ) $
 +
is not absolutely continuous with respect to  $  p _  \xi  \times p _  \eta  ( \cdot ) $,
 +
then  $  I ( \xi , \eta ) = + \infty $,
 +
by definition.
 +
 
 +
In case the random variables  $  \xi $
 +
and  $  \eta $
 +
take only a finite number of values, the expression for  $  I ( \xi , \eta ) $
 +
takes the form
 +
 
 +
$$
 +
I ( \xi , \eta )  = \sum _{i=1} ^ { n }  \sum _{j=1} ^ { m }
 +
p _ {ij}  \mathop{\rm log} \
 +
 
 +
\frac{p _ {ij} }{p _ {i} q _ {i} }
 +
,
 +
$$
  
 
where
 
where
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105028.png" /></td> </tr></table>
+
$$
 +
\{ p _ {i} \} _ {i=} 1  ^ {n} ,\ \
 +
\{ q _ {j} \} _ {j=} 1  ^ {m} ,\ \
 +
\{ {p _ {ij} } : {i = 1 \dots n ; j = 1 \dots m } \}
 +
$$
  
are the probability functions of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105029.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105030.png" /> and the pair <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105031.png" />, respectively. (In particular,
+
are the probability functions of $  \xi $,  
 +
$  \eta $
 +
and the pair $  ( \xi , \eta ) $,
 +
respectively. (In particular,
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105032.png" /></td> </tr></table>
+
$$
 +
I ( \xi , \xi )  = -
 +
\sum _{i=1} ^ { n }
 +
p _ {i}  \mathop{\rm log}  p _ {i}  = H ( \xi )
 +
$$
  
is the [[Entropy|entropy]] of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105033.png" />.) In case <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105034.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105035.png" /> are random vectors and the densities <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105036.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105037.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105038.png" /> of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105039.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105040.png" /> and the pair <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105041.png" />, respectively, exist, one has
+
is the [[entropy]] of $  \xi $.)  
 +
In case $  \xi $
 +
and $  \eta $
 +
are random vectors and the densities $  p _  \xi  ( x) $,  
 +
$  p _  \eta  ( y) $
 +
and $  p _ {\xi \eta }  ( x , y ) $
 +
of $  \xi $,  
 +
$  \eta $
 +
and the pair $  ( \xi , \eta ) $,  
 +
respectively, exist, one has
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105042.png" /></td> </tr></table>
+
$$
 +
I ( \xi , \eta )  = \
 +
\int\limits p _ {\xi \eta }  ( x , y )  \mathop{\rm log} 
 +
\frac{p _ {\xi \eta }
 +
( x , y ) }{p _  \xi  ( x) p _  \eta  ( y) }
 +
\
 +
d x  d y .
 +
$$
  
 
In general,
 
In general,
  
<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105043.png" /></td> </tr></table>
+
$$
 +
I ( \xi , \eta )  = \
 +
\sup  I ( \phi ( \xi ) , \psi ( \eta ) ) ,
 +
$$
  
where the supremum is over all measurable functions <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105044.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/i/i051/i051050/i05105045.png" /> with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.
+
where the supremum is over all measurable functions $  \phi ( \cdot ) $
 +
and $  \psi ( \cdot ) $
 +
with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.
  
 
For references, see , ,
 
For references, see , ,
  
 
to [[Information, transmission of|Information, transmission of]].
 
to [[Information, transmission of|Information, transmission of]].

Latest revision as of 16:27, 6 January 2024


An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let $ \xi $ and $ \eta $ be random variables defined on a probability space $ ( \Omega , \mathfrak A , {\mathsf P} ) $ and taking values in measurable spaces (cf. Measurable space) $ ( \mathfrak X , S _ {\mathfrak X } ) $ and $ ( \mathfrak Y , S _ {\mathfrak Y } ) $, respectively. Let $ p _ {\xi \eta } ( C) $, $ C \in S _ {\mathfrak X } \times S _ {\mathfrak Y } $, and $ p _ \xi ( A) $, $ A \in S _ {\mathfrak X } $, $ p _ \eta ( B) $, $ B \in S _ {\mathfrak Y } $, be their joint and marginale probability distributions. If $ p _ {\xi \eta } ( \cdot ) $ is absolutely continuous with respect to the direct product of measures $ p _ \xi \times p _ \eta ( \cdot ) $, if $ a _ {\xi \eta } ( \cdot ) $ is the (Radon–Nikodým) density of $ p _ {\xi \eta } ( \cdot ) $ with respect to $ p _ \xi \times p _ \eta ( \cdot ) $, and if $ i _ {\xi \eta } ( \cdot ) = \mathop{\rm log} a _ {\xi \eta } ( \cdot ) $ is the information density (the logarithms are usually taken to base 2 or $ e $), then, by definition, the amount of information is given by

$$ I ( \xi , \eta ) = \ \int\limits _ {\mathfrak X \times \mathfrak Y } i _ {\xi \eta } ( x , y ) p _ {\xi \eta } ( d x , d y ) = $$

$$ = \ \int\limits _ {\mathfrak X \times \mathfrak Y } a _ {\xi \eta } ( x , y ) \mathop{\rm log} \ a _ {\xi \eta } ( x , y ) p _ \xi ( d x ) p _ \eta ( d y ) . $$

If $ p _ {\xi \eta } ( \cdot ) $ is not absolutely continuous with respect to $ p _ \xi \times p _ \eta ( \cdot ) $, then $ I ( \xi , \eta ) = + \infty $, by definition.

In case the random variables $ \xi $ and $ \eta $ take only a finite number of values, the expression for $ I ( \xi , \eta ) $ takes the form

$$ I ( \xi , \eta ) = \sum _{i=1} ^ { n } \sum _{j=1} ^ { m } p _ {ij} \mathop{\rm log} \ \frac{p _ {ij} }{p _ {i} q _ {i} } , $$

where

$$ \{ p _ {i} \} _ {i=} 1 ^ {n} ,\ \ \{ q _ {j} \} _ {j=} 1 ^ {m} ,\ \ \{ {p _ {ij} } : {i = 1 \dots n ; j = 1 \dots m } \} $$

are the probability functions of $ \xi $, $ \eta $ and the pair $ ( \xi , \eta ) $, respectively. (In particular,

$$ I ( \xi , \xi ) = - \sum _{i=1} ^ { n } p _ {i} \mathop{\rm log} p _ {i} = H ( \xi ) $$

is the entropy of $ \xi $.) In case $ \xi $ and $ \eta $ are random vectors and the densities $ p _ \xi ( x) $, $ p _ \eta ( y) $ and $ p _ {\xi \eta } ( x , y ) $ of $ \xi $, $ \eta $ and the pair $ ( \xi , \eta ) $, respectively, exist, one has

$$ I ( \xi , \eta ) = \ \int\limits p _ {\xi \eta } ( x , y ) \mathop{\rm log} \frac{p _ {\xi \eta } ( x , y ) }{p _ \xi ( x) p _ \eta ( y) } \ d x d y . $$

In general,

$$ I ( \xi , \eta ) = \ \sup I ( \phi ( \xi ) , \psi ( \eta ) ) , $$

where the supremum is over all measurable functions $ \phi ( \cdot ) $ and $ \psi ( \cdot ) $ with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.

For references, see , ,

to Information, transmission of.

How to Cite This Entry:
Information, amount of. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Information,_amount_of&oldid=12464
This article was adapted from an original article by R.L. DobrushinV.V. Prelov (originator), which appeared in Encyclopedia of Mathematics - ISBN 1402006098. See original article