Partial Derivatives and Gradient Vectors

I finished the last practice problem in the machine learning fundamentals course today. The lesson moved from the steepest direction in two dimensions, through local minima and maxima, into higher-dimensional derivatives and why derivatives matter in machine learning. I got stuck for a bit around partial derivatives and gradient vectors. The idea was not impossibly complicated, but the picture in my head became blurry once slope moved from one variable to several variables.

With a single-variable function, slope is fairly easy to imagine. I can look at how much $y$ changes when $x$ moves a little. If the instantaneous rate of change is positive, the function value goes up as $x$ increases; if it is negative, the function value goes down as $x$ increases.

The point where the slope is zero needed a bit more care. A zero slope only means the graph is flat at that instant, not that the point is automatically a minimum or maximum. If the graph goes down, touches the bottom, and comes back up, that point is a local minimum. If it goes up, touches the top, and comes back down, that point is a local maximum. The smallest point across the whole range is the global minimum, and the largest one is the global maximum.

A saddle point is the case that still feels a little strange. The slope can look like zero there, but the point is neither a bottom nor a top. In one direction it can look like the surface is going down, while in another direction it can look like it is going up. The name makes sense once I imagine an actual saddle shape.

Then the input variables increase to two.

f(x, y) = x^2 + 2y^2

Now the function value is no longer decided by $x$ alone. Both $x$ and $y$ go into the function, and together they determine the height. If I think of it like a map, one axis is east-west, the other is north-south, and the function value is close to the altitude at that location.

A partial derivative is a way to move only one variable at a time. When differentiating with respect to $x$ , I temporarily hold $y$ fixed; when differentiating with respect to $y$ , I hold $x$ fixed. If both move at the same time, it becomes hard to separate which variable changed the function value by how much.

So when I partially differentiate $f(x, y) = x^2 + 2y^2$ with respect to $x$ , the result is $2x$ . The $2y^2$ part only contains $y$ , so from the point of view of $x$ , it is treated like a constant. Since a constant has no rate of change, it disappears.

\frac{\partial f(x, y)}{\partial x} = 2x

When differentiating with respect to $y$ , I treat $x^2$ like a constant and only differentiate $2y^2$ .

\frac{\partial f(x, y)}{\partial y} = 4y

A new symbol showed up here. Or maybe I had seen it before and forgot. Instead of the usual $d$ , partial derivatives use the curled symbol $\partial$ . With a single-variable function, $\frac{dy}{dx}$ means differentiating $y$ with respect to $x$ , and the $dx$ in the denominator marks $x$ as the reference variable for the rate of change. Partial derivatives are similar, but because the function has several inputs, the notation has to make clear which one variable I am using as the reference at that moment.

So $\frac{\partial f}{\partial x}$ means I keep the other variables fixed and change only $x$ a tiny amount, then look at how much $f$ changes. $\frac{\partial f}{\partial y}$ does the same thing with $y$ while keeping $x$ fixed. For a single-variable function, the usual phrasing is “differentiate with respect to $x$ .” For a function with several inputs, it becomes “take the partial derivative with respect to $x$ ” or “take the partial derivative with respect to $y$ .”

If I collect the partial derivatives for each variable, I get a gradient vector.

\nabla f(x, y) = \begin{bmatrix} 2x \\ 4y \end{bmatrix}

I also looked up the upside-down triangle symbol and found that it is called nabla. For now, I am reading it as a way to collect the rate of change in each direction. It puts the steepness in the $x$ direction and the steepness in the $y$ direction into one vector.

For example, if I plug in $(x, y) = (1, 1)$ , I get this.

\nabla f(1, 1) = \begin{bmatrix} 2 \\ 4 \end{bmatrix}

At first, the 2 and 4 did not really land. After unpacking it, 2 means how much the function value changes when I keep $y = 1$ fixed and move $x$ slightly. The 4 means how much the function value changes when I keep $x = 1$ fixed and move $y$ slightly.

Using the map analogy, if my current location is $(1, 1)$ , moving one step east raises the altitude by roughly 2. Moving one step north raises it by roughly 4. In that sense, it is easier to see which direction is steeper. At this point, the change in the $y$ direction is larger than the change in the $x$ direction.

But the gradient vector does not stop at saying “east is 2, north is 4.” It combines those two pieces and points to the steepest upward direction from the current location. The vector $(2, 4)$ moves in both the $x$ and $y$ directions, but more strongly toward $y$ , so that seems to be why the gradient vector is described as the direction where the function increases the fastest.

If I want to go down as fast as possible, I can put a minus sign in front of it.

-\nabla f(1, 1) = \begin{bmatrix} -2 \\ -4 \end{bmatrix}

I understood this as saying that if I want to reduce the function value from the current position, I should move in the opposite direction of the gradient vector. The last quiz said that a slope can point toward either an increasing direction or a decreasing direction. My current understanding is that the gradient itself tells me the increasing direction, and the opposite direction becomes the decreasing one.

I also connected this to why derivatives are needed in machine learning. To evaluate a model’s performance, I need some number that tells me whether things are getting better or worse, and I also need to know which direction makes that number go down.