01.6 Prediction, Learning and Cross-Entropy Loss

We explore the role of entropy in prediction and learning problems

Example — Classify Images

For every image $x$, our Neural Network $\text{NN}$ outputs a probability distribution over all possible labels $l$

$$Q_{\text{NN}}(l\mid x)$$

For every image $x$, the true label distribution is

$$P_{\text{true}}(l\mid x)=\{\begin{array}{ll}1,& l\text{ is the correct label for }x\\ 0,& \text{wrong label}\end{array}$$

Ideally, we want (for every pair image-label)

$$Q_{\text{NN}}(l\mid x)=P_{\text{true}}(l\mid x)$$

→ This will never happen !

Instead, we consider cross entropy loss

Concretely, for every image $x$, we wish to minimize the loss

$$L(P_{\text{true}}(\cdot \mid x),Q_{\text{NN}}(\cdot \mid x))=-\sum _{l=1}^K{P}_{\text{true}}(l\mid x){\log }_D{Q}_{\text{NN}}(l\mid x)$$

Definition

Cross Entropy Loss between two distributions $P$ and $Q$

$$L(P,Q)=-\sum _{y}P(y){\log }_{D}Q(y)$$

Theorem

For a fixed probability distribution $P$, the minimum

$$\underset{Q}{\min }L(P,Q)$$

Is attained iff we select $Q^{\ast }=P$, and in this case would be given by

$$L(P,Q^{\ast })=L(P,P)=H(P)$$