Linear Regression and Loss Functions

Today I studied linear regression and loss functions through a house price prediction example. As the lecture moved forward, the main point became evaluating which line is better with a number.

Suppose I want to predict the price of a 30-pyeong apartment. If I already have data that pairs house size with house price, I can fit a straight line through those points and use that line to predict a new price. This is the line of best fit, and linear regression is the process of finding that line and using it for prediction.

In this example, house size is the input variable, also called a feature. House price is the target variable, or output variable. The value I want to predict is the price, and the information used to make that prediction is the size.

Machine learning is often divided into supervised learning, unsupervised learning, and reinforcement learning. Linear regression belongs to supervised learning because the training data includes both inputs and answers. The model is not only given house sizes; it is also given the corresponding prices.

The lecture also introduced classification and regression. Classification chooses one category from a fixed set, such as apartment, house, or officetel. Regression predicts a continuous value, such as a house price.

I now separate them this way: if the output is a category, it is classification; if the output is a continuously changing numeric value, it is regression. A number alone does not automatically make a task regression. The criterion is whether the value moves continuously as a prediction target. House price does, so it is a regression problem.

In linear regression, the line is written as a hypothesis function. In the familiar form, it looks like this.

y = ax + b

The notation itself is not the point yet. The important part is that different lines produce different predictions, so I need a way to measure how far each line’s predictions are from the actual values.

That is where mean squared error, or MSE, comes in. If the prediction exactly matches the real value, the error is 0. In practice, most predictions are off by some amount. MSE takes those differences, squares them, and averages them.

\text{MSE} = \frac{1}{m}\sum_{i=1}^{m}\left(h(x^{(i)}) - y^{(i)}\right)^2

Here, $m$ is the number of training examples. $x^{(i)}$ is the $i$ -th input value, $y^{(i)}$ is the actual answer for that input, and $h(x^{(i)})$ is the prediction made by the current hypothesis function.

Squaring the error does two things. First, positive and negative errors do not cancel each other out. Second, large errors receive a larger penalty. An error of 2 becomes 4, while an error of 10 becomes 100, so a prediction that is far away from the real value is treated as much worse.

MSE is a number that says how much the line misses the data overall. A smaller value means the line fits the data better, and a larger value means the predictions are farther from the actual values.

After MSE, the lecture moved from $y = ax + b$ to theta notation. Instead of writing the slope and intercept as $a$ and $b$ , machine learning notation usually writes the hypothesis function like this.

h_\theta(x) = \theta_0 + \theta_1 x

Here, $h_\theta(x)$ means the value predicted by the model when input $x$ is given. In the house price example, $x$ is the house size, and $h_\theta(x)$ is the predicted house price.

$\theta_1$ is the slope of the line. It decides how much the predicted price changes when the house size increases by 1. $\theta_0$ is the constant term. It is the value that remains when $x$ is 0, and on the graph it decides where the line crosses the vertical axis. I understood $\theta_0$ as the value that shifts the whole prediction line up or down.

So the goal of linear regression is to find useful theta values. Changing $\theta_0$ and $\theta_1$ changes the position and slope of the line, which then changes the prediction for the same input.

When a straight line is not enough

A single straight line may not describe data whose input and target follow a curved relationship. Polynomial regression handles this by adding powers of the original input, such as $x^2$ and $x^3$ , as new features.

A second-degree polynomial regression model can be written as:

h_\theta(x)=\theta_0+\theta_1x+\theta_2x^2

The $x^2$ term makes the graph curve, but the parameters $\theta_0$ , $\theta_1$ , and $\theta_2$ are still combined linearly. Polynomial regression therefore remains a form of linear regression. It transforms the input into polynomial features and learns their weights with the same underlying method.

The naming became confusing here. Adding more input variables also adds more terms to the equation, so multiple linear regression and polynomial regression can initially look like two names for the same thing.

Their equations can look similar, but the names describe where the terms came from, not simply how many terms the equation contains.

Model	Original input	Features given to the model
Simple Linear Regression	House size	House size
Multiple Linear Regression	House size, room count	House size, room count
Polynomial Regression	House size	House size, house size²
Multiple Polynomial Regression	House size, room count	House size, room count, house size², house size × room count

Multiple linear regression gains terms by adding different input variables, such as house size and room count. I now use multiple to mean that the model has more than one input variable. Polynomial regression instead creates new features from existing inputs, such as the square of house size. When a model has multiple input variables and also includes powers or products of those inputs, it can be called multiple polynomial regression. Multiple describes the number of original inputs, while polynomial describes the degree to which those inputs have been expanded.

Polynomial regression first creates a feature such as $x^2$ from house size $x$ . The model then receives two values, house size and house size squared, multiplies them by $\theta_1$ and $\theta_2$ , and adds the constant $\theta_0$ to produce a prediction. This is the same calculation structure used by multiple linear regression when it multiplies house size and room count by separate weights and adds them.

This also explains why a polynomial regression curve still belongs to the linear regression family. Linear does not require $x$ to remain first degree. It means that the learned coefficients $\theta_0$ , $\theta_1$ , and $\theta_2$ are not multiplied by one another or squared; each coefficient simply multiplies one feature, and the resulting terms are added.

The word polynomial caused one more point of confusion. In mathematics, first-degree expressions are also polynomials. In machine learning, however, a first-degree model already has the familiar name linear regression, so polynomial regression usually distinguishes models that add terms such as $x^2$ or higher powers. It is simply called polynomial regression rather than polynomial linear regression.

It also helps to separate a term from the degree of a polynomial.

Term	Meaning	Example
Term	One part of an expression separated by addition or subtraction	$\theta_1x$
Quadratic term	A term whose variable exponents add up to 2	$\theta_2x^2$ , $\theta_3x_1x_2$
Interaction term	A term formed by multiplying different input variables	$\theta_3x_1x_2$
Degree	The highest sum of exponents among the terms	$\theta_0+\theta_1x+\theta_2x^2$ has degree 2

A square is not the only possible quadratic term. In $x_1x_2$ , each variable has exponent 1, so their exponents add up to 2. It is a second-degree interaction term because it represents the combined contribution of two inputs. Both $\theta_2x^2$ and $\theta_3x_1x_2$ are quadratic terms, and the full expression is second degree if it contains no higher-degree term.

Why create polynomial features?

Adding several input variables does not automatically represent an extra effect that appears only when two conditions occur together. Consider this simplified apartment-price dataset:

Apartment	Large $x_1$	Prime location $x_2$	Actual price $y$
Small, outer area	0	0	200 million won
Large, outer area	1	0	300 million won
Small, prime location	0	1	1 billion won
Large, prime location	1	1	1.8 billion won

Here, $x_1$ and $x_2$ are 1 when the condition is present and 0 otherwise. The model receives these inputs and the actual price $y$ , then learns weights that reduce the loss between its predicted and actual prices.

An additive equation without an interaction term can represent the first three prices like this:

\hat{y}=2+1x_1+8x_2

The equation adds 100 million won for a large apartment and 800 million won for a prime location. It therefore predicts 1.1 billion won for the apartment that is both large and in a prime location, missing the additional premium contained in the actual price of 1.8 billion won. Changing the existing weights to fit that final apartment would also change the other three predictions, so this additive structure cannot fit all four prices at once.

An interaction term provides a separate value that applies only when both conditions are present:

\hat{y}=2+1x_1+8x_2+7x_1x_2

$x_1x_2$ equals 1 only for an apartment that is both large and in a prime location. The first three prices remain unchanged, while the final apartment receives the additional 700 million won needed to represent its actual price. Polynomial features are useful because they let the equation represent relationships that cannot be explained by independent input effects alone.

While an interaction term represents a combination of two inputs, a squared term such as $x^2$ represents a curved relationship between one input and the result. Polynomial regression adds squared and interaction features when a straight line or a purely additive equation cannot describe the data.

Increasing the degree allows the model to represent more complicated curves, but it can also make the model follow small variations in the training data and overfit. Choosing the degree is therefore a model-complexity decision, which also connects to feature scaling and regularization.

A loss function calculates how far the hypothesis function’s predictions are from the actual values. In linear regression, mean squared error is used as that loss value.

The lecture wrote the loss function as $J$ .

J(\theta) = \frac{1}{2m}\sum_{i=1}^{m}\left(h_\theta(x^{(i)}) - y^{(i)}\right)^2

The loss function is almost the same as MSE, except that the denominator has an extra 2. I now separate the two ideas like this: MSE is the way the error is calculated, and the loss function returns that error value for the current theta.

Another confusing point was why the input of the loss function is $\theta$ , not $x$ or $y$ . The formula clearly contains $x$ and $y$ , so why is it written as $J(\theta)$ instead of $J(x, y)$ ?

The explanation depends on treating the training data as fixed. The house sizes $x$ and actual prices $y$ are already given. What I can change is not the data, but the shape of the hypothesis function, and that shape is determined by $\theta_0$ and $\theta_1$ .

The output of the loss function therefore changes when theta changes. With the same data, a different theta gives a different line, that line gives different predictions, and those predictions produce a different MSE value. That is why the loss function is written as $J(\theta)$ .

The coefficient also changed from $\frac{1}{m}$ to $\frac{1}{2m}$ . Strictly speaking, this is MSE multiplied by $\frac{1}{2}$ , so the numeric value is cut in half. However, the theta that minimizes the value does not change. The lecture explained that the extra 2 is there to make later calculations cleaner.

At this point, I understand linear regression as finding the best-fit line, and the loss function as the numeric way to evaluate that line. The input data and answers are already fixed, and the values being adjusted are theta values. Among many possible theta values, linear regression tries to find the ones that make the loss function as small as possible.

I have not covered the concrete method for reducing the loss yet. For this note, it is enough to connect MSE to the loss function and understand why the loss function takes theta as its input.