Difference between revisions of "Information, amount of"
(Importing text file) |
(latex details) |
||
| (One intermediate revision by one other user not shown) | |||
| Line 1: | Line 1: | ||
| − | + | <!-- | |
| + | i0510501.png | ||
| + | $#A+1 = 45 n = 0 | ||
| + | $#C+1 = 45 : ~/encyclopedia/old_files/data/I051/I.0501050 Information, amount of | ||
| + | Automatically converted into TeX, above some diagnostics. | ||
| + | Please remove this comment and the {{TEX|auto}} line below, | ||
| + | if TeX found to be correct. | ||
| + | --> | ||
| − | + | {{TEX|auto}} | |
| + | {{TEX|done}} | ||
| − | + | An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let $ \xi $ | |
| + | and $ \eta $ | ||
| + | be random variables defined on a [[Probability space|probability space]] $ ( \Omega , \mathfrak A , {\mathsf P} ) $ | ||
| + | and taking values in measurable spaces (cf. [[Measurable space|Measurable space]]) $ ( \mathfrak X , S _ {\mathfrak X } ) $ | ||
| + | and $ ( \mathfrak Y , S _ {\mathfrak Y } ) $, | ||
| + | respectively. Let $ p _ {\xi \eta } ( C) $, | ||
| + | $ C \in S _ {\mathfrak X } \times S _ {\mathfrak Y } $, | ||
| + | and $ p _ \xi ( A) $, | ||
| + | $ A \in S _ {\mathfrak X } $, | ||
| + | $ p _ \eta ( B) $, | ||
| + | $ B \in S _ {\mathfrak Y } $, | ||
| + | be their joint and marginale probability distributions. If $ p _ {\xi \eta } ( \cdot ) $ | ||
| + | is absolutely continuous with respect to the direct product of measures $ p _ \xi \times p _ \eta ( \cdot ) $, | ||
| + | if $ a _ {\xi \eta } ( \cdot ) $ | ||
| + | is the (Radon–Nikodým) density of $ p _ {\xi \eta } ( \cdot ) $ | ||
| + | with respect to $ p _ \xi \times p _ \eta ( \cdot ) $, | ||
| + | and if $ i _ {\xi \eta } ( \cdot ) = \mathop{\rm log} a _ {\xi \eta } ( \cdot ) $ | ||
| + | is the information density (the logarithms are usually taken to base 2 or $ e $), | ||
| + | then, by definition, the amount of information is given by | ||
| − | + | $$ | |
| + | I ( \xi , \eta ) = \ | ||
| + | \int\limits _ {\mathfrak X \times \mathfrak Y } | ||
| + | i _ {\xi \eta } ( x , y ) p _ {\xi \eta } ( d x , d y ) = | ||
| + | $$ | ||
| − | + | $$ | |
| + | = \ | ||
| + | \int\limits _ {\mathfrak X \times \mathfrak Y } a _ {\xi \eta } ( x , y ) \mathop{\rm log} \ | ||
| + | a _ {\xi \eta } ( x , y ) p _ \xi ( d x ) p _ \eta ( d y ) . | ||
| + | $$ | ||
| − | + | If $ p _ {\xi \eta } ( \cdot ) $ | |
| + | is not absolutely continuous with respect to $ p _ \xi \times p _ \eta ( \cdot ) $, | ||
| + | then $ I ( \xi , \eta ) = + \infty $, | ||
| + | by definition. | ||
| + | |||
| + | In case the random variables $ \xi $ | ||
| + | and $ \eta $ | ||
| + | take only a finite number of values, the expression for $ I ( \xi , \eta ) $ | ||
| + | takes the form | ||
| + | |||
| + | $$ | ||
| + | I ( \xi , \eta ) = \sum _{i=1} ^ { n } \sum _{j=1} ^ { m } | ||
| + | p _ {ij} \mathop{\rm log} \ | ||
| + | |||
| + | \frac{p _ {ij} }{p _ {i} q _ {i} } | ||
| + | , | ||
| + | $$ | ||
where | where | ||
| − | + | $$ | |
| + | \{ p _ {i} \} _ {i=} 1 ^ {n} ,\ \ | ||
| + | \{ q _ {j} \} _ {j=} 1 ^ {m} ,\ \ | ||
| + | \{ {p _ {ij} } : {i = 1 \dots n ; j = 1 \dots m } \} | ||
| + | $$ | ||
| − | are the probability functions of | + | are the probability functions of $ \xi $, |
| + | $ \eta $ | ||
| + | and the pair $ ( \xi , \eta ) $, | ||
| + | respectively. (In particular, | ||
| − | + | $$ | |
| + | I ( \xi , \xi ) = - | ||
| + | \sum _{i=1} ^ { n } | ||
| + | p _ {i} \mathop{\rm log} p _ {i} = H ( \xi ) | ||
| + | $$ | ||
| − | is the [[ | + | is the [[entropy]] of $ \xi $.) |
| + | In case $ \xi $ | ||
| + | and $ \eta $ | ||
| + | are random vectors and the densities $ p _ \xi ( x) $, | ||
| + | $ p _ \eta ( y) $ | ||
| + | and $ p _ {\xi \eta } ( x , y ) $ | ||
| + | of $ \xi $, | ||
| + | $ \eta $ | ||
| + | and the pair $ ( \xi , \eta ) $, | ||
| + | respectively, exist, one has | ||
| − | + | $$ | |
| + | I ( \xi , \eta ) = \ | ||
| + | \int\limits p _ {\xi \eta } ( x , y ) \mathop{\rm log} | ||
| + | \frac{p _ {\xi \eta } | ||
| + | ( x , y ) }{p _ \xi ( x) p _ \eta ( y) } | ||
| + | \ | ||
| + | d x d y . | ||
| + | $$ | ||
In general, | In general, | ||
| − | + | $$ | |
| + | I ( \xi , \eta ) = \ | ||
| + | \sup I ( \phi ( \xi ) , \psi ( \eta ) ) , | ||
| + | $$ | ||
| − | where the supremum is over all measurable functions | + | where the supremum is over all measurable functions $ \phi ( \cdot ) $ |
| + | and $ \psi ( \cdot ) $ | ||
| + | with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission. | ||
For references, see , , | For references, see , , | ||
to [[Information, transmission of|Information, transmission of]]. | to [[Information, transmission of|Information, transmission of]]. | ||
Latest revision as of 16:27, 6 January 2024
An information-theoretical measure of the quantity of information contained in one random variable relative to another random variable. Let $ \xi $
and $ \eta $
be random variables defined on a probability space $ ( \Omega , \mathfrak A , {\mathsf P} ) $
and taking values in measurable spaces (cf. Measurable space) $ ( \mathfrak X , S _ {\mathfrak X } ) $
and $ ( \mathfrak Y , S _ {\mathfrak Y } ) $,
respectively. Let $ p _ {\xi \eta } ( C) $,
$ C \in S _ {\mathfrak X } \times S _ {\mathfrak Y } $,
and $ p _ \xi ( A) $,
$ A \in S _ {\mathfrak X } $,
$ p _ \eta ( B) $,
$ B \in S _ {\mathfrak Y } $,
be their joint and marginale probability distributions. If $ p _ {\xi \eta } ( \cdot ) $
is absolutely continuous with respect to the direct product of measures $ p _ \xi \times p _ \eta ( \cdot ) $,
if $ a _ {\xi \eta } ( \cdot ) $
is the (Radon–Nikodým) density of $ p _ {\xi \eta } ( \cdot ) $
with respect to $ p _ \xi \times p _ \eta ( \cdot ) $,
and if $ i _ {\xi \eta } ( \cdot ) = \mathop{\rm log} a _ {\xi \eta } ( \cdot ) $
is the information density (the logarithms are usually taken to base 2 or $ e $),
then, by definition, the amount of information is given by
$$ I ( \xi , \eta ) = \ \int\limits _ {\mathfrak X \times \mathfrak Y } i _ {\xi \eta } ( x , y ) p _ {\xi \eta } ( d x , d y ) = $$
$$ = \ \int\limits _ {\mathfrak X \times \mathfrak Y } a _ {\xi \eta } ( x , y ) \mathop{\rm log} \ a _ {\xi \eta } ( x , y ) p _ \xi ( d x ) p _ \eta ( d y ) . $$
If $ p _ {\xi \eta } ( \cdot ) $ is not absolutely continuous with respect to $ p _ \xi \times p _ \eta ( \cdot ) $, then $ I ( \xi , \eta ) = + \infty $, by definition.
In case the random variables $ \xi $ and $ \eta $ take only a finite number of values, the expression for $ I ( \xi , \eta ) $ takes the form
$$ I ( \xi , \eta ) = \sum _{i=1} ^ { n } \sum _{j=1} ^ { m } p _ {ij} \mathop{\rm log} \ \frac{p _ {ij} }{p _ {i} q _ {i} } , $$
where
$$ \{ p _ {i} \} _ {i=} 1 ^ {n} ,\ \ \{ q _ {j} \} _ {j=} 1 ^ {m} ,\ \ \{ {p _ {ij} } : {i = 1 \dots n ; j = 1 \dots m } \} $$
are the probability functions of $ \xi $, $ \eta $ and the pair $ ( \xi , \eta ) $, respectively. (In particular,
$$ I ( \xi , \xi ) = - \sum _{i=1} ^ { n } p _ {i} \mathop{\rm log} p _ {i} = H ( \xi ) $$
is the entropy of $ \xi $.) In case $ \xi $ and $ \eta $ are random vectors and the densities $ p _ \xi ( x) $, $ p _ \eta ( y) $ and $ p _ {\xi \eta } ( x , y ) $ of $ \xi $, $ \eta $ and the pair $ ( \xi , \eta ) $, respectively, exist, one has
$$ I ( \xi , \eta ) = \ \int\limits p _ {\xi \eta } ( x , y ) \mathop{\rm log} \frac{p _ {\xi \eta } ( x , y ) }{p _ \xi ( x) p _ \eta ( y) } \ d x d y . $$
In general,
$$ I ( \xi , \eta ) = \ \sup I ( \phi ( \xi ) , \psi ( \eta ) ) , $$
where the supremum is over all measurable functions $ \phi ( \cdot ) $ and $ \psi ( \cdot ) $ with a finite number of values. The concept of the amount of information is mainly used in the theory of information transmission.
For references, see , ,
Information, amount of. Encyclopedia of Mathematics. URL: http://encyclopediaofmath.org/index.php?title=Information,_amount_of&oldid=12464