From Loss Functions to Gradient Descent

In the previous note, I wrote about linear regression and loss functions. Using the house price example, I understood linear regression as finding a line that explains the data, and the loss function as the function that gives a numeric score for how wrong that line is.

This time, I wanted to connect that setup to gradient descent. If a loss function evaluates the current line, gradient descent is the method that changes the line little by little so the loss goes down. It is easier to understand when the order stays intact: hypothesis function, loss function, partial derivatives, gradient vector, and then gradient descent.

---

In linear regression, the hypothesis function is a line. In the familiar form, it is written as this.

$$y = ax + b$$

In machine learning notation, the same idea is usually written with theta.

$$h_\theta(x) = \theta_0 + \theta_1 x$$

Here, $x$ is the input data. In the house price example, it is the house size. $h_\theta(x)$ is the price predicted by the current line, and $\theta_0$ and $\theta_1$ decide the position and slope of that line.

The training data, $x$ and $y$, is already given. What I can change is not the data itself but the shape of the line, and that shape is controlled by $\theta_0$ and $\theta_1$. So learning in linear regression means adjusting those theta values so the loss becomes smaller.

---

To decide whether the current theta values are good or bad, I need a score. That score is computed by the loss function. In linear regression, the loss function is based on mean squared error and is written like this.

$$J(\theta) = \frac{1}{2m}\sum_{i=1}^{m}\left(h_\theta(x^{(i)}) - y^{(i)}\right)^2$$

The formula loops over the training data, compares the prediction from the current hypothesis function with the actual value, squares the difference, and averages it. A smaller value means the current line explains the data better. A larger value means the predictions are farther away.

The important shift for me was separating the two graphs. At first, I kept thinking about the two-dimensional graph of house size and house price. But the loss function is a different graph. The house size $x$ and actual price $y$ are fixed data, while the values that move inside the loss function are $\theta_0$ and $\theta_1$.

The hypothesis-function graph takes $x$ and returns a predicted price. The loss-function graph takes $\theta_0$ and $\theta_1$ and returns a loss value. What I want to reduce is not the predicted price itself, but the loss, so the function I need to differentiate is $J(\theta)$.

---

To reduce the loss, I need to know which direction to move from the current point. I could try every possible combination of $\theta_0$ and $\theta_1$, but that stops being practical as the range grows. Instead, I calculate how the loss function changes around the current theta values.

This is where partial derivatives come back. The loss function $J(\theta)$ changes with both $\theta_0$ and $\theta_1$. If I move both at the same time, it is hard to separate which one affected the loss and by how much.

When differentiating with respect to $\theta_0$, I temporarily treat $\theta_1$ as fixed. When differentiating with respect to $\theta_1$, I treat $\theta_0$ as fixed. That lets me see how the loss changes when each parameter moves slightly on its own.

$$\frac{\partial J}{\partial \theta_0}, \quad \frac{\partial J}{\partial \theta_1}$$

These values are the slopes of the loss function in each parameter direction at the current point. One tells me how the loss changes if I move in the $\theta_0$ direction, and the other tells me how the loss changes if I move in the $\theta_1$ direction.

---

When those partial derivatives are collected together, they form the gradient vector.

$$\nabla J(\theta) = \begin{bmatrix} \dfrac{\partial J}{\partial \theta_0} \\[0.8em] \dfrac{\partial J}{\partial \theta_1} \end{bmatrix}$$

This vector points in the direction where the loss function increases the fastest from the current point. It is the uphill direction of the loss surface. But I do not want the loss to increase. I want it to decrease, so I move in the opposite direction. That is gradient descent.

$$\theta_0^{new} = \theta_0^{old} - \alpha \frac{\partial J}{\partial \theta_0}$$ $$\theta_1^{new} = \theta_1^{old} - \alpha \frac{\partial J}{\partial \theta_1}$$

Here, $\alpha$ is the learning rate. The partial derivative tells me the direction, and the learning rate decides how far to move in one update. If it is too small, the loss decreases slowly. If it is too large, the update can overshoot the minimum and become unstable.

I have not reached the point of implementing this in code yet, so I want to keep the note at the level of the formula. The update subtracts the partial derivative, scaled by the learning rate, from the old theta value and uses the result as the new theta value. In other words, it moves the parameter slightly in the direction that reduces the loss.

Repeating this update changes $\theta_0$ and $\theta_1$ little by little. As theta changes, the hypothesis line changes. As the line changes, the predictions change, and the loss is calculated again. Gradient descent is this repeated update process for moving the parameters in the direction that reduces the loss.

---

The part I needed to separate most clearly was the graph being differentiated. There is the hypothesis-function graph for house size and house price, and there is the loss-function graph for theta and loss. Gradient descent differentiates the second one, not the first one. In the simple linear regression example, that loss surface has three dimensions: loss, $\theta_0$, and $\theta_1$. As the number of input variables grows, the number of theta values also grows, so the dimension grows with it.

The overall flow now looks like this to me. The hypothesis function makes a prediction, and the loss function calculates how wrong that prediction is. Then I partially differentiate the loss function with respect to each theta value to find the local direction of change, collect those values into the gradient vector, and move theta a little in the opposite direction. That updates the line toward a lower loss.

I have not gone into manually expanding the derivative for real data or choosing a learning rate yet. For now, I understand gradient descent as the procedure that connects the earlier pieces together: repeatedly updating parameters to reduce the value of the loss function.