View source for Kullback-Leibler information

''Kullback–Leibler quantity of information, Kullback–Leibler information quantity, directed divergence''

For discrete distributions (cf. [[Discrete distribution|Discrete distribution]]) given by probability vectors <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101801.png" />, <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101802.png" />, the Kullback–Leibler (quantity of) information of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101803.png" /> with respect to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101804.png" /> is:

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101805.png" /></td> </tr></table>

where <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101806.png" /> is the natural logarithm (cf. also [[Logarithm of a number|Logarithm of a number]]).

More generally, one has:

<table class="eq" style="width:100%;"> <tr><td valign="top" style="width:94%;text-align:center;"><img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101807.png" /></td> </tr></table>

for probability distributions <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101808.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k1101809.png" /> with densities <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018010.png" /> and <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018011.png" /> (cf. [[Density of a probability distribution|Density of a probability distribution]]).

The negative of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018012.png" /> is the conditional entropy (or relative entropy) of <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018013.png" /> with respect to <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018014.png" />; see [[Entropy|Entropy]].

Various notions of (asymmetric and symmetric) information distances are based on the Kullback–Leibler information.

The quantity <img align="absmiddle" border="0" src="https://www.encyclopediaofmath.org/legacyimages/k/k110/k110180/k11018015.png" /> is also called the informational divergence (see [[Huffman code|Huffman code]]).

See also [[Information distance|Information distance]]; [[Kullback–Leibler-type distance measures|Kullback–Leibler-type distance measures]].

====References====
<table><TR><TD valign="top">[a1]</TD> <TD valign="top">  S. Kullback,   "Information theory and statistics" , Wiley  (1959)</TD></TR><TR><TD valign="top">[a2]</TD> <TD valign="top">  S. Kullback,   R.A. Leibler,   "On information and sufficiency"  ''Ann. Math. Stat.'' , '''22'''  (1951)  pp. 79–86</TD></TR><TR><TD valign="top">[a3]</TD> <TD valign="top">  J. Sakamoto,   M. Ishiguro,   G. Kitagawa,   "Akaike information criterion statistics" , Reidel  (1986)</TD></TR></table>