The cross entropy of the distribution relative to a distribution over a given set is defined as follows: where is the expected value operator with respect to the distribution. The definition may be formulated using the Kullback–Leibler divergence from of . where is the entropy of. For discrete probability distributions and with the same support this means The situation for continuous distributions is analogous. We have to assume that and are absolutely continuous with respect to some reference measure . Let and be probability density functions of and with respect to. Then and therefore NB: The notation is also used for a different concept, the joint entropy of and.
Motivation
In information theory, the Kraft-McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value out of a set of possibilities can be seen as representing an implicit probability distribution over, where is the length of the code for in bits. Therefore, cross entropy can be interpreted as the expected message-length per datum when a wrong distribution is assumed while the data actually follows a distribution. That is why the expectation is taken over the true probability distribution and not. Indeed the expected message-length under the true distribution is,
Estimation
There are many situations where cross-entropy needs to be measured but the distribution of is unknown. An example is language modeling, where a model is created based on a training set, and then its cross-entropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, is the true distribution of words in any corpus, and is the distribution of words as predicted by the model. Since the true distribution is unknown, cross-entropy cannot be directly calculated. In these cases, an estimate of cross-entropy is calculated using the following formula: where is the size of the test set, and is the probability of event estimated from the training set. The sum is calculated over. This is a Monte Carlo estimate of the true cross entropy, where the test set is treated as samples from.
Relation to log-likelihood
In classification problems we want to estimate the probability of different outcomes. If the estimated probability of outcome is, while the frequency of outcome in the training set is, and there are N conditionally independent samples in the training set, then the likelihood of the training set is so the log-likelihood, divided by is so that maximizing the likelihood is the same as minimizing the cross entropy.
Cross-entropy minimization
Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution against a fixed reference distribution, cross entropy and KL divergence are identical up to an additive constant : both take on their minimal values when, which is for KL divergence, and for cross entropy. In the engineering literature, the principle of minimising KL Divergence is often called the Principle of Minimum Cross-Entropy, or Minxent. However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution is the fixed prior reference distribution, and the distribution is optimised to be as close to as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be, rather than.
Cross entropy can be used to define a loss function in machine learning and optimization. The true probability is the true label, and the given distribution is the predicted value of the current model. More specifically, consider logistic regression, which can be used to classify observations into two possible classes. The output of the model for a given observation, given a vector of input features, can be interpreted as a probability, which serves as the basis for classifying the observation. The probability is modeled using the logistic function where is some function of the input vector, commonly just a linear function. The probability of the output is given by where the vector of weights is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output is simply given by Having set up our notation, and, we can use cross entropy to get a measure of dissimilarity between and : Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. For example, suppose we have samples with each sample indexed by. The average of the loss function is then given by: where, with the logistic function as before. The logistic loss is sometimes called cross-entropy loss. It is also known as log loss.
