Log Loss and Cross-Entropy

This post continues from my earlier notes on logistic regression and decision boundaries. Quite some time had passed since the class, and much of it had faded, so I went through the related Codeit lessons again and looked up the parts I could no longer reconstruct from memory. There are more formulas here than usual. I am writing them down now so I have something concrete to return to as my understanding develops.

Measuring Errors in Probability Predictions

A classification model calculates probabilities before selecting a class. How should it distinguish between predicting 0.9 and 0.1 when the correct label is 1? Using logistic regression as the main example, this post examines how Log Loss and Cross-Entropy measure errors in probability predictions.

Logistic regression passes a linear score $z$ through the sigmoid function to produce a probability $\hat{y}$ between 0 and 1. The true label $y$ is either 0 or 1, and the loss function should become smaller as the model assigns more probability to the correct label.

Why Cross-Entropy Instead of MSE?

Linear regression can use Mean Squared Error by squaring the difference between the prediction and the actual value. Logistic regression produces a probability through the sigmoid function, while the target remains either 0 or 1. Its loss should be small when the model assigns a high probability to the correct answer and grow sharply when the model is confidently wrong.

When the correct label is 1, the loss is:

-\log(\hat{y})

As $\hat{y}$ approaches 1, the loss approaches 0. As $\hat{y}$ approaches 0, the loss grows without bound. This does not mean that the loss grows exponentially because a logarithm is involved. The negative logarithm diverges as $\hat{y}$ approaches 0, assigning a large penalty when the correct label is 1 but the model predicts a probability near 0.

When the correct label is 0, the loss is:

-\log(1-\hat{y})

Combining both cases gives Binary Cross-Entropy, or BCE.

\ell(\hat{y},y) = -\left[ y\log(\hat{y}) +(1-y)\log(1-\hat{y}) \right]

When $y=1$ , the second term disappears, leaving only $-\log(\hat{y})$ . When $y=0$ , the first term disappears, leaving $-\log(1-\hat{y})$ . One formula therefore handles both possible labels.

For binary classification, BCE is also called Log Loss. The expression above is the loss for one example, while the full cost is the average across all $m$ examples:

J(w,b) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)}\log\hat{y}^{(i)} + (1-y^{(i)})\log(1-\hat{y}^{(i)}) \right]

Combining the sigmoid output with MSE can produce a non-convex cost function with respect to the weights, which makes the minimum harder to find with gradient descent. For linear logistic regression, BCE produces a convex cost function with respect to the model parameters. It also has a probabilistic interpretation: training increases the likelihood assigned to the observed labels.

A convex cost function has one overall bowl-shaped structure, so optimization does not become trapped in an inferior local minimum. Linear regression with MSE is also convex, but its derivative can be rearranged into the Normal Equation and solved directly with matrix operations. Logistic regression contains the sigmoid function, so its derivative does not allow the weight $w$ to be isolated in the same way. Even though its BCE cost is convex, an iterative method such as gradient descent is needed to find the minimum.

Learning the Parameters by Reducing the Loss

Logistic regression can use the same gradient descent process covered earlier. The model calculates a probability with the current $w$ and $b$ , evaluates it with BCE, and repeatedly updates the parameters in the direction that reduces the cost.

calculate the linear score z
  → convert it into a probability with sigmoid
  → calculate the loss with BCE
  → update w and b with gradient descent
  → use the learned w and b as the decision boundary

Gradient descent learns $w$ and $b$ by reducing BCE. The final parameters define the decision boundary:

w^{\mathsf{T}}x+b=0

I covered the gradient descent process itself in an earlier post.

From Loss Functions to Gradient Descent →

Multiclass Classification and Softmax

Basic logistic regression separates two classes. For red, blue, and green, a one-vs-rest strategy trains red versus the rest, blue versus the rest, and green versus the rest as separate binary classifiers. The class with the highest score or probability is selected.

Softmax converts every class score $z_k$ into a probability at once:

p_k=\frac{e^{z_k}}{\sum_j e^{z_j}}

The class probabilities sum to 1, and the class with the highest probability becomes the prediction. Training uses multiclass Cross-Entropy to increase the probability of the correct class, and this loss is also minimized with gradient descent.

Cross-Entropy is not limited to logistic regression. Binary classification commonly combines a sigmoid output with BCE, while single-label multiclass classification commonly combines Softmax with multiclass Cross-Entropy. In both cases, training reduces the loss by assigning more probability to the correct class.

I had initially mixed up convexity with the ability to use the Normal Equation. Convexity describes a cost function whose minimum can be found without competing local minima; it does not mean that the solution can be calculated in one matrix operation. Logistic regression still uses an iterative optimizer such as gradient descent to reach that minimum.

The previous post made logistic regression itself much clearer, but connecting it to loss functions and Cross-Entropy took longer. Even when my understanding is incomplete, writing down a first version feels more efficient than leaving the topic untouched. The unfinished parts give me a concrete reason to return, review the material, and revise the explanation until it matches what I understand.