Linear Regression and Loss Functions

Today I studied linear regression and loss functions through a house price prediction example. As the lecture moved forward, the main point became evaluating which line is better with a number.

Suppose I want to predict the price of a 30-pyeong apartment. If I already have data that pairs house size with house price, I can fit a straight line through those points and use that line to predict a new price. This is the line of best fit, and linear regression is the process of finding that line and using it for prediction.

In this example, house size is the input variable, also called a feature. House price is the target variable, or output variable. The value I want to predict is the price, and the information used to make that prediction is the size.

Machine learning is often divided into supervised learning, unsupervised learning, and reinforcement learning. Linear regression belongs to supervised learning because the training data includes both inputs and answers. The model is not only given house sizes; it is also given the corresponding prices.

The lecture also introduced classification and regression. Classification chooses one category from a fixed set, such as apartment, house, or officetel. Regression predicts a continuous value, such as a house price.

I now separate them this way: if the output is a category, it is classification; if the output is a continuously changing numeric value, it is regression. A number alone does not automatically make a task regression. The criterion is whether the value moves continuously as a prediction target. House price does, so it is a regression problem.

In linear regression, the line is written as a hypothesis function. In the familiar form, it looks like this.

y = ax + b

The notation itself is not the point yet. The important part is that different lines produce different predictions, so I need a way to measure how far each line’s predictions are from the actual values.

That is where mean squared error, or MSE, comes in. If the prediction exactly matches the real value, the error is 0. In practice, most predictions are off by some amount. MSE takes those differences, squares them, and averages them.

\text{MSE} = \frac{1}{m}\sum_{i=1}^{m}\left(h(x^{(i)}) - y^{(i)}\right)^2

Here, $m$ is the number of training examples. $x^{(i)}$ is the $i$ -th input value, $y^{(i)}$ is the actual answer for that input, and $h(x^{(i)})$ is the prediction made by the current hypothesis function.

Squaring the error does two things. First, positive and negative errors do not cancel each other out. Second, large errors receive a larger penalty. An error of 2 becomes 4, while an error of 10 becomes 100, so a prediction that is far away from the real value is treated as much worse.

MSE is a number that says how much the line misses the data overall. A smaller value means the line fits the data better, and a larger value means the predictions are farther from the actual values.

After MSE, the lecture moved from $y = ax + b$ to theta notation. Instead of writing the slope and intercept as $a$ and $b$ , machine learning notation usually writes the hypothesis function like this.

h_\theta(x) = \theta_0 + \theta_1 x

Here, $h_\theta(x)$ means the value predicted by the model when input $x$ is given. In the house price example, $x$ is the house size, and $h_\theta(x)$ is the predicted house price.

$\theta_1$ is the slope of the line. It decides how much the predicted price changes when the house size increases by 1. $\theta_0$ is the constant term. It is the value that remains when $x$ is 0, and on the graph it decides where the line crosses the vertical axis. I understood $\theta_0$ as the value that shifts the whole prediction line up or down.

So the goal of linear regression is to find useful theta values. Changing $\theta_0$ and $\theta_1$ changes the position and slope of the line, which then changes the prediction for the same input.

A loss function calculates how far the hypothesis function’s predictions are from the actual values. In linear regression, mean squared error is used as that loss value.

The lecture wrote the loss function as $J$ .

J(\theta) = \frac{1}{2m}\sum_{i=1}^{m}\left(h_\theta(x^{(i)}) - y^{(i)}\right)^2

The loss function is almost the same as MSE, except that the denominator has an extra 2. I now separate the two ideas like this: MSE is the way the error is calculated, and the loss function returns that error value for the current theta.

Another confusing point was why the input of the loss function is $\theta$ , not $x$ or $y$ . The formula clearly contains $x$ and $y$ , so why is it written as $J(\theta)$ instead of $J(x, y)$ ?

The explanation depends on treating the training data as fixed. The house sizes $x$ and actual prices $y$ are already given. What I can change is not the data, but the shape of the hypothesis function, and that shape is determined by $\theta_0$ and $\theta_1$ .

The output of the loss function therefore changes when theta changes. With the same data, a different theta gives a different line, that line gives different predictions, and those predictions produce a different MSE value. That is why the loss function is written as $J(\theta)$ .

The coefficient also changed from $\frac{1}{m}$ to $\frac{1}{2m}$ . Strictly speaking, this is MSE multiplied by $\frac{1}{2}$ , so the numeric value is cut in half. However, the theta that minimizes the value does not change. The lecture explained that the extra 2 is there to make later calculations cleaner.

At this point, I understand linear regression as finding the best-fit line, and the loss function as the numeric way to evaluate that line. The input data and answers are already fixed, and the values being adjusted are theta values. Among many possible theta values, linear regression tries to find the ones that make the loss function as small as possible.

I have not covered the concrete method for reducing the loss yet. For this note, it is enough to connect MSE to the loss function and understand why the loss function takes theta as its input.