Cross-Entropy Loss

Cross-entropy is the standard loss for classification — including the next-token prediction that trains language models. It measures the distance between what the model predicted and what actually happened.

The formula

For a single example with true class y and predicted probability p for the correct class:

L = −log(p)

For the full distribution over classes:

L = − Σ  y_i · log(p_i)

Because the true label is usually one-hot (y_i is 1 for the right class, 0 elsewhere), the sum collapses to just −log(p_correct).

Why the log matters

The −log term is the whole personality of this loss:

• Predict the right answer with p = 0.99 → loss ≈ 0.01 (barely penalized).
• Predict it with p = 0.5 → loss ≈ 0.69 (meaningful nudge).
• Predict it with p = 0.01 → loss ≈ 4.6 (enormous penalty).

So a model that is confidently wrong is punished far more than one that is merely uncertain. This pressure pushes models toward well-calibrated probabilities.

The connection back to people

There’s a learning-science echo here: confident errors are also the most valuable ones for humans to correct — the “hypercorrection effect,” where high-confidence mistakes, once corrected, are the least likely to recur. Both systems learn most from being confidently wrong and finding out.